Securing a cloud native open source microservice based core banking system- Ntando Mngomezulu

foreign

good morning everyone it is still morning yeah all right so uh names in tando I'm about to have a you know do a talk on cloud security securing a core banking system in AWS I don't know if I should kick it off or let people settle in or but I'm gonna start it off actually let's do just to get the ball rolling all right so like any other talk the best place to start is to give you a little bit about me and yeah so comsi graduate who's pretty much just a family guy you know wife and a kid actually I got married last week so yeah I'm Shackled down now um yeah chess enthusiasts I actually

participated in the world's first chess.com World Championship and I sucked I I didn't make it past the open qualifiers but I will try again next year um before I move on from chess actually this year was a good year for chess you know as I said chess.com open World Championships but also uh Magnus Carson and the whole butt plug thing no you guys you guys know nothing about it yeah uh for anybody who doesn't know anything about it just have it chuckled the two of them yeah so I spend most of my days you know Jason the gentleman who did the previous talk I think Mr Spencer he spends most of his days breaking in I

spend most of my days keeping him up um so I'm on the blue side of things and yeah another interesting thing about me I'm an open source evangelist so if you give me proprietary software side by side with open source software I'll almost always choose the open source so yeah human me um just want to get to know a little bit about the crowd the audience please do humor me and participate in this hands up if you regularly work inside or work in AWS okay keep your hands up don't take them down hands up or keep your hands up and take your hands down keep them up if you use kubernetes regularly nice keep your hands up still if you've

deployed a core banking system in kubernetes I like that mice I like that that's pretty cool it's actually quite crazy were you at hexcon not it's it's always one guy in the audience that ends up keeping his hand up through the whole questions okay so about the talk yeah we're going to be just discussing you know like I said initially the security of the cloud which I think is a very important topic and just to set the problem statement I've got two slides that'll kind of show how important this is and why we need to take this seriously the first slide I'm not going to go through all those points you can read them for yourselves

the first slide just shows that 10 of the world's I.T infrastructure has migrated to the cloud now that's not such a big number right yeah ten percent who cares you know but juxtapose that with this next slide which says Cloud security breaches have surpassed on-prem breaches for the first time research done by Verizon 2021 tail end of 2021 how is it that 10 of the world's infrastructure is being breached more than the other 90 that's a big those are two big facts there so so for people like me you blue teamers even the red teams I'm not gonna I'm not gonna take anything away from the importance of what you do we need to take this seriously you know

um because you don't want to be another Facebook you want to be another LinkedIn you know you hear it all the time another one got hacked today blah blah blah but it's up to us to keep that from happening and hence topics like this just to spread their awareness spread the knowledge all right so that's that's jump into this AWS humble Origins I think way back when started off as just a collection of you know simple storage service which is still currently its Flagship ec2 the compute service um and that's still today those two Services comprise most of what we use AWS for um but yeah I'm not going to talk too much on AWS for now I want to Pivot away

and just kind of discuss the core banking system for a little bit on the left we have a gentleman that I you know I look up to this guy I Revere this guy actually Dr Mohammed yunus he is a Noble Peace prizing holding gentleman highly esteemed gentleman for those of you who know in the audience he's also known as the father of microfinance or micro credit so just to quickly Sprint through like what he's done here and just the origins of of finraxi and finnure Cloud native um the gentleman started a Grameen bank and then went on to start the Crimean foundation in addition to other Grameen you know subsidiaries green and and and During the period of around 2004 and

2007 and before I before I go on what I love about green bank and what speaks to me on this slide is the fact that you know look at the title it says bank for the poor and that speaks to financial inclusivity being on the African continent that really touches the court in my heart you know so so the origins of finraxia and so to speak kind of seep through you know the code speaks to this this motto this Vision this mission statement so yeah 2004 and 2007 there was like a a burst in commercial activity in the back it grew quite substantially and during that period they began working on finnurexcn Under the Umbrella of myfos x

that got handed Over The myfos X Project got handed over to the Grameen Foundation under the myfos initiative which then handed the project over under the thinner Acts incubator program in 2011 it then subsequently graduated as a top level program in 2017. now that's a major note of approval from Apache just speaks to the maturity of the software and and then somewhere in the mix there finneraxi and Kmart finereact mifos X is a monolith then they broke that down into microservices in order to make it Cloud radio and Cloud native all right so let's speak a little bit more about these micro Services the building blocks of this core banking system so if you have a look there what we have

is the set of services at the bottom under so you have you know member Administration which and then you have membership notifications verify ID finrack services including Office provisioner Accounting deposit and and customer one of the more important microservices in this diagram is the identity microservice which is the authentication server for finrax CN which is underpinned by a system called Anubis but as you can see very simple diagram coming in from the left from the front end hitting the Gateway and then going through to the microservices where finally information is stored in the data store another picture describing the same thing but differently we have all the microservices and and your nosql and SQL Cassandra and

postgres respectively storing all the data all right so just as an FYI to those who actually want to go and check the software out the software is quite I wouldn't say Limited does what it's supposed to do but if you have to bring the software into your own context you know you have to build upon it and expand upon it and this is kind of a slide that speaks to that where you had additional micro services that worked in tandem with that set of core banking systems in order to expand upon for the context in which that software runs member Administration notifications and verify ID for those of you who don't know verify ID it's a real-time interface into the

natural person's registry which is just home Affairs allowing you to verify in real time you know whether someone is who they say they are and that's for the you know kyc process for those of you who are in banking all right Okay so I don't want to belabor too much the point on on finner xdn so we'll move on from there and begin jumping into you know more of the architectural level discussion kind of dissecting um the system in a greater context um no Cloud talk is complete without a short description of the shared responsibility model and the catchphrase goes AWS or your cloud service provider is responsible for security of the cloud and we as analysts and Engineers are

responsible for security in the cloud this this you know concept comes in later on when we're kind of unpacking the kubernetes side of things as well but it permeates almost every aspect of your interaction with AWS so over here we have that in a more you know tangible View at the bottom you've got your you know AWS Services your regions your azs your availability zones your compute and Storage you know S3 EBS Etc and your networking vpcs etc etc which allow whatever workflows you're putting on AWS to achieve your business objectives and then at the top that's where we're responsible for security and you know whether it's secure encryption of data at rest encryption of data and Transit

configuration of those systems we play at the top AWS plays at the bottom another kind of diagram that shows this but with a little bit more context specific information is this one which shows our co-banking system I don't know if this is really legible for your reading guys but I am apologize I apologize I can't zoom in unfortunately so we have the core banking system we've got the member Administration couple of you know the gravity Gateway et cetera Etc everything above the foundational Services is for us to configure and then everything below is for AWS to to worry about all right now this brings me to what I call the golden thread I've just quickly had a brief

Sprint through the architectural aspects of the system that we'll be kind of discussing and now the golden thread of what I call the security landscape in AWS you see on the left you've got IAM you've got single sign-on you know and then you've got Cloud Watch Inspector etc etc what we're going to be doing as we go along is we're going to be ticking boxes off on this landscape as we go through each section and try hit as many boxes as we can in order to ensure a secure environment foreign again not very legible but I will zoom in this time for you this is you know I would say a very basic architectural diagram you guys have seen far more

complex diagrams and but this just serves a purpose just to emphasize what I was speaking about in terms of the the golden thread that I showed you just now so you see traffic coming in from the corporate offices via the VPN you also see traffic coming in from the internet into the environment most importantly I want to bring your attention to the first thing that the end user hits which is a load balancer that's associated to a ref and has a cert attached to it for encryption of of traffic in transit via the AWS certificate manager again very very basic public subnet private subnet nothing complex but just emphasizing a point here and if we go a

little bit further we see at the back we've got our eks cluster with an auto scaling group that you know spins up nodes or spins them down accordingly and then right at the bottom we've got our CI CD pipeline that's you know got Version Control and Jenkins you know speaking to each other so now if you if you are paying attention to the golden thread previously we see we're already starting to tick a couple of boxes we see the waft being ticked off we've got a VPC being ticked off cert manager VPN AWS Shield so already you know the security of your infrastructure starts architecturally and that's the that's the that's the idea here to say let's

design a secure infrastructure before you even touch a line of code before you even configure a single service you know foreign

this brings me to the section on securing the stack so in this section I'm actually going to there's a nice there's a nice way to think about securing the cloud 4cs right securing the cloud no matter what cloud service provider you're using which is the slide we just came from curing the cluster securing the container that's from a kubernetes perspective and then securing the code the four C's so we're going to start off with securing the cloud the previous slide and the next upcoming slide will speak to that as well and let's just jump straight into it so hands up for anybody who has ever worked with the control tower or deploy The Landing Zone nice nice all right so

um just a way to explain this a control tower is a way to provision a multi-account environment right for AWS now what that does for me the the best way to think about it is you wouldn't want to put you know an FTP server mail server DNS server all on one box it becomes too much of a risk in terms of the number of services that you're exposing on it this is a similar concept you're trying to split up multiple different aspects of your environment into logically separate areas logically separate accounts so you know in security we've got our core tenants right you've got the principle of least privilege you've also got the principle of need to know think

of this as the principle of separation of concerns right so you don't want to have one account being too high risk that account being compromised in the whole thing going down so yeah let's let's jump into dissecting exactly what this diagram shows in the top account the root account we have the control tower and organizations control tower is just your orchestration for this multi-year account Landing Zone whereas the organizations allows you to manage it's the UI the interface for the administrator to manage this multi-account Landings on coming to how we authenticate into the landing Zone we have the directory service on AWS and then we have single sign-on what I see most companies do most companies that I've

interacted with actually because I mean that's the that's the AWS native version of the service usually most people come in with their 365 as your active directory and replace that top icon with the with the aad and then after that the users inside that active directory will then use single sign-on on AWS to authenticate in and then this is a very very important section in terms of the architecture that I'm showing you here if you look we have multiple organizational units uh one for you know each specific section note the asterisks on the applications OU which just basically means you know multiple of those so multiplicity and you'll have whether it's your uat mat sit product etc etc as

as many as as you want from from a business perspective but right at the end on the uh end there you have the SCP the service control policies and this is one of the key aspects to securing a uh Landing Zone it is an organizational level uh way of saying what services can and cannot do which services are allowed and which aren't allowed and inside each service what can be done and what cannot be done paying attention to this is critical all right so going into our application accounts maybe that's Dev prod whatever the case may be we again we see service control policies except this time on a per account basis but what we also see

is right at the bottom there IAM policies so the combination of organizational level uh scps account level scps and IIM policies that's the try factor that really is the the beating hearts of security in an LZ so to speak controlling these three is Paramount to ensuring that you've got a grip and a lockdown on this LZ all right so moving to the networking account again just speaking to how we're separating the concerns so that if one of the vpcs is compromised not all of them are compromised and that also means not the entire infrastructure has been taken down giving you time to respond from an incidence response perspective before the guy pivots and takes down

more of your system so we see a couple of vpcs egress shared VPC client VPN VPC those are quite self-explanatory the egress VPC allowing people traffic are allowing systems traffic out of the network the shared VPC for all the OU's hence the asterisks on it and then the client VPN VPC allowing VPN clients to enter the network okay last but not least we've got the audience account the audit account is actually quite a key account because it contains you know all your keys for you know the things that you'll be encrypting for example you'll have an EBS volume that requires encryption so you'll have encryption at rest and you will store the key to encrypt

and decrypt that volume inside your KMS key store and that key store lives inside the audit account making this an extremely sensitive account I've seen I've actually seen LZ deployments where the KMS key store is taken out of the audit account and put into an account called the crypto account however you know refined or granular you want to make it is up to you but the key points don't put the key store in other systems where for example your VPC resources will reside so for those of you have been paying attention you'll notice these security Hub and guard Duty icons in every single account and [Music] if we go down inside the audit account we'll have the central guard Duty and

the Central Security Hub account so effectively you're having all those guard Duty which is threat intelligence effectively and security Hub which is compliance management coming into this Central account for Auditors and consultants and analysts to review and act upon those upon that information all right so this is a I think quite a an important structure if you haven't dealt with an LZ before dig into it it's quite important and yeah but let's move on and as you can see the golden thread we've ticked quite a lot of the boxes in the identity and access management column again I'm sorry for the legibility and at the end there you see control tower being ticked and you've got KMS

and yeah and VPC flow logs also being ticked there so a couple of more boxes being ticked in our security service landscape on AWS and it's important when you're securing an environment to bring especially one in AWS to bring as many of the tools that you have at your disposal to perform against real threat actors all right so this brings me to this section of the talk which um what I would call the core security offering that is provided on AWS so you have a combination of your guard Duty again threat Intelligence coming in from you know various logs VPC flow logs SD API logs etc etc and then you've also got your security Hub which is your

compliance in this picture unfortunately I forgot to include PCI that is one of the compliance standards I'm sure many of you actually deal with that regularly but we've also got the AWS foundational standard on the sys compliance all of that coming in and hooked into some sort of alert system that allows you to see these alerts as they're generated as and when they're generated this is the you know again a very core basic model for security on AWS so let's expand upon it you have on your right couple of new services you've got Macy which hunts down poppy information you know it's it's so easy to to slip and miss you know a data store or a repository of

information that's got a lot of client data and it's happened I've seen it happen you want the machines to help you out with that um then you've got your IIM access analyzer you know are there outdated keys that need rotating are there any users that need to be removed because they haven't been accessed accessed in a long time then you've got your SSM patch manager now to be honest SSM is the systems manager is an amazing tool it's more than just a patch manager for example recently we had to configure a weft right and and you know you've got a couple of ec2 instances in a client environment you don't really know what's running on them so we deployed an SSM

agent onto these machines that kind of scrubbed the system and told us what software was installed you know node PHP etc etc created an inventory of that and then that allows us to then go back to the web and refine the rule sets recording you don't want to just throw all the rules on the web that really don't make sense you know if it's a Windows box you don't want to put Linux rule sets you know so so that's quite a neat feature of the of the security manager and actually it goes much deeper than that but patching is one of the big things that you want to use that for and then you've got inspector which you know

we used quite recently to to scan for the log4 shell vulnerability across all uh containers that were that were running in in multiple environments okay now as Security Consultants Defenders attackers you know one thing we don't want to be doing is adding to the problem you know if you're going to be configuring a security architecture like this you don't want to be the guy that allows the guy the other the bad guys in and S3 really is one of the ways which a lot of companies have found their pants down because of misconfigurations all right now so you might be asking me how does this architecture fit into the previous architecture which I just shown

which is the landing zone right notice emphasized the god Duty elements of each account and you're seeing God Duty and insecurity up here so indeed what you have is you've got the main audit account feeding security Hub information in from all the child accounts and that is the architecture that we'll be going for when securing the environment right so that brings me to the golden thread yet again and notice now this time how quite a lot of the boxes are now finally ticked off unfortunately um and Ivan I don't know if Ivan's here but unfortunately this column here is something where you know I was at the hex-con conference and Ivan Ivan Birch I think

mentioned there's such an under supply of incidence responders in South Africa you know and I I'm proving that right now in preparation of my talk I didn't even I didn't even touch that that column there but for the most part you know you're seeing the light starting to switch on across across the the security service landscape all right that brings me to the end of architecturally how you would go about securing you know your your Cloud environment let's jump into the kubernetes environment where the code will be running so a couple of you guys already know kubernetes I kind of asked that question at the beginning what you'll notice if you're paying attention I will come back

to this now actually I'm getting ahead of myself so let's just start with a brief overview of how kubernetes is put together so you have your API server which is one of the more critical components in the architecture that's what the Kube CTL command line application interacts with when an administrator kind of interacts with the cluster all right and then you've got your Cloud controller manager the icon at the top there and then you've got your hcd hcd being well before I go to NCD Cloud controller manager being the API to the you know cloud service providers resources and then you've got your SCD which is a distributed High availability data store which stores the state of the cluster

now compromise of either the API server or the hcd data store is akin to root on a Linux box so that's that's what you want to protect right at the beginning of the offset

yeah I'm already thirsty now so all right so then we've got our nodes recall in the previous architectural diagram the simple diagram that we had that auto scaling group it's actually responsible for spinning up the nodes which are just regular ec2 instances inside those nodes what you'll have running is the kubelet um and the kublet via the container runtime interface manages the pods inside your nodes and then you've got the coupe proxy which is the physical implementation of a service in kubernetes now for the eagle-eyed audience member you'll notice there are two elements missing here there's the regular controller manager the controller manager I'm not going to get into too much detail here but has it manages controllers

one of them if you ever want to get into kubernetes research one of them that you'd really want to look at is the admission controller it before a part is scheduled it will then mutate and verify certain elements of that pot or that manifests before scheduling it on pot and then you've got another thing that's missing here is the scheduler actually says okay cool you've given me a put Let me find a note to run it on and then and then the kublet will say okay I've got I've got a pot now let me find a container let me put the container inside the pot so those two systems the scheduler and the kubernet work hand in hand to deploy the

workflows that you'll be that you'll be using to uh to run your systems all right another view that I'd like to kind of give you is the flow of network traffic right so you've got your red line coming from the admin to the API server very very important to protect that then you've got your end user coming in through the service you know that'll be an HTTP endpoint that they'll be interacting with and then you've got your container registry where the Pod pulls its containers down from the internet from assuming it's a public container registry all right now let's discuss some of the attacker entry points I've already mentioned that you know the API server as well as the hcd server

are high value targets those need protection at all costs then you'll have your service I mean anything that the end user can get to the hacker can get to as well right and then you've got your node and your Kubler over here this there's some actually really neat vulnerabilities out there one being a container Escape phone on the crio runtime most runtimes today actually I think the majority of of kubernetes clusters run on Docker but coupler really need crio vulnerabilities out there for their own researchers and then you've got your registry which stores the images which will end up running as containers all right so let's begin just unpacking um you know how each of these elements

can be protected to keep you know threat actor out before I go any further for those of you who do work in the environment you'll notice that I've kind of cheated I've added those two blocks there and if you're on a AWS cluster you'll never actually see the control plane you won't see the master node it's invisible to you that ties back to you know the shared responsibility model that's something AWS is in charge of you know securing however that being said there is a certain degree of control that we do have at this point here which is control over the API server so let's jump into that so what you have is most of the time you want to have a

rollback system protecting that but in this case AWS implements that rollback system via IAM you don't have to worry too much about that what you can worry about is who has access to the API server if you make that API server public what you'd want to do is you know whitelist certain trusted IP addresses that can access that API server the preferred method the recommended method is making your API server private and then using perhaps a jump box or a Transit gateway to connect to that to that to that private API endpoint again so I'm going to move on to the next slide but as I move on to the next slide you'll notice I've put an x on API

server I've also put an x on hcd datastore you cannot control anything on the hcd data store that's AWS so hence that was just for information purposes all right so the Pod security policy what you want to do is when you're running a put inside a node you want to control what level of access that Port has to the underlying host whether it's got host ports access whether it's got host network access etc etc that's done via the you know pod security policy as well as in combination with the network policy I'm so sorry that this can't be read properly but I hope I'm narrating it properly for you guys um and one important thing to know about

the Pod security policy is that for those who deal with this again regularly this is deprecated as of version 1.21 of kubernetes and I think it'll be completely phased out end of life by 1.25 and then I'd like to pause for a second and have a Shameless plug um and have a little bit of a question error to see who wants these you know synthesis goodies I think there's a couple of shirts in here key holders socks chocolate cups for anybody who can answer the next question for me which is and I'll I'll give it to the person who gives me the most correct answer I'm not sure if everybody's going to get it right but it is easy

um so we're moving away from the Pod security policy to protecting the registry anybody want to have a shot at how you protect your register your hands up no is there nice okay that's actually there you go anybody else want to give it a shot I mean there's a couple of things maybe we split them up nobody the gift's yours man so a notary server right signatures but in addition to static code analyzers and actually something very interesting a friend of mine showed me earlier on this week is chat GPT in terms of secure code reviews or static code reviews have a look at that but yeah so a notary server you'd wanna you'd wanna you know

there's two options here that you want to play with one you're gonna go with third-party hosted notary server the problem with that is that you're not going to have all the signatures you know you're gonna you're gonna you're gonna take what you get the other alternative is you know creating your own privately owned notary server which then leaves you with the job of creating those signatures yourself and then of course you know open source all proprietary scans of those images using a static code analyzer all right okay protecting the service like I said earlier on the same way an attacker can get to the uh the back end the same way an end user can get to the back end is

the same way an attacker does and how I've had success in the past is converting the service to an Ingress of type nginx and then enabling the mod security rule sets with the O wasp core rule set on there so the mod security plug-in excuse me with the OS core rule set now you might ask yourself but Nintendo that's just the duplication of the weft that you showed me right at the beginning indeed but if you look a little bit deeper and for those who have played with nginx you know you've got a lot of control over some of the security HTTP headers and that gives you control over what the attacker can and cannot do so

that's a very very cool combination there that you can use to protect the service and then last but not least you have the AWS resources that are sitting on the cloud such as your abs block or block storage what you want to do here is you want to have an IAM policy attached to a service account that you force the Pod to run with so that pod cannot access what you haven't given it access to even if it gets compromised right so so that's a that's a pretty neat one to block that attack Vector as well I don't know how I'm doing for time okay looks like I'm good so yeah um and that brings me to the end guys of

you know securing kubernetes which also means it's the end of the stack I did say that there are four C's I'm not a coder so I'm not gonna lie and stand here and tell you about how to secure the code you know I'll tell you about how to secure the infrastructure but I do know that secure code reviews does get you a long way again I'll name drop that that thing again chat GPT check it out in terms of you know the feature of secure code reviews but yeah that's the uh that's the end of what I have to show you basically I think the the take home here for me and I hope this is what you're

taking home is you know a multi-layered approach to security which is the you know which is the standard you start off with securing the cloud the cluster the container and at each level you wanna you wanna observe certain core tenants of security like the security uh the principle of separation of concerns which we saw on the LZ and uh this is the principle of least privilege which we'll see when we're associating a policy to a service account but other than that That's All She Wrote guys that's all she wrote uh thank you for your time

do we have any questions awesome go first sir

yeah so that's actually an interesting question we've been working on 40s very recently for East West North South traffic inspection on AWS so yeah it's something that I'm coming up across in the real world and yeah you point out a good point the fact that I don't have it in my slides but it is something that you'd want to look at but yeah I think next time around with version two of these slides I'll add it in there thanks so much 100 thank you

cool cool nice thanks where are you gonna ask yeah so

actually no because we we found we found it difficult to get God Duty into kubernetes so we ended up using you know open up proprietary software which I don't like to always lean on we use Prisma specifically Swiss lock kind of to to get to gain insight into into the cluster

yeah okay so there was a Trend Micro conference that we just recently attended with the team and so instance metadata service version 2 imds V2 right there's a couple of server-side requests forgery attacks that can go on it and we got that feedback from God Duty in the Island region so we're sitting there trying to secure this cluster from this imds V2 attack and we're noticing wait hold on if soft doesn't scan for that finding so there was an AWS representative at the Trend Micro conference and I'm saying to him so you're saying South African security admins have to fight this fight with their hands tied behind our base and and then he just left okay okay I'll get

back to you you know and and but it was a valid question because you're running scans in Ireland which pick up this very bad vulnerability same scans in Cape Town the the Cape Town region do not what does that mean for people like us who have to protect information that resides in South Africa because government wants the information for our citizens to stay here poppy right so yeah so that's a that's a big one that's a very very big one I don't know if I answered your question or if I went on a tangent there okay cool yeah yeah 100 I've seen them I've seen them in practice and I'm speaking to AWS and I'm telling them you should ask fix it

you know and and yeah they're trying they're trying I guess okay cool

yeah yeah yeah not 100 100 I agree with you man I agree with you any other questions

from a yeah

you say Falco okay cool is that a service mesh type of

okay okay cool thanks I'll remember that Falco Falco thanks I'll remember that man thanks

yeah yeah

so you know how I'm going to answer your question is I've I spoke to a couple of csos and I'm like you know companies getting hacked csos getting sacked it's a it's a shame you know when a company gets hacked and then the ciso gets sacked you know um so the answer to to your question is how far do you want to go you can look at it from a commercial perspective you're paying more for that peace of mind and you're offloading that risk that you know you no longer responsible for if it gets hacked but how I would look at it you know and but you you're playing premium for that for that peace

of mind you know because now you know that seesaw who's spending more money than he should be but you know that if somebody gets into your network you get to point the finger you know so that's another way of looking at it you know

yeah yeah

um so SSM is pretty cool now for that you can you can kind of associate into it using your browser without having to expose it uh so you run the instance you put it you put an agent on the SSM agent on there and you'll be able to interact with the terminal via the browser without having to you know set up keys and things like that um and using putty or whatever to connect over the internet you'll just be you know HTTP your way into that so yeah so that's that's kind of without revealing too much about how we do what we do that's that's a good way to do it but there are other ways to you know

skim that cat as well I'm running out of water guys I'll better up any other questions thank you so much