GF - Lessons Learned by the WordPress Security Team - Aaron D Campbell

please welcome Aaron Campbell thank you so I'm Aaron Campbell and I lead the security team for the WordPress open-source project I've been doing that for a little over a year and a half but been a part of the team basically since we started it and we've learned a lot of lessons along the way most of them that I'm going over today are not specific coding technical things it's a lot more of how our team functions well and how we are able to run security for that large of a project I am employed by GoDaddy to work full time on WordPress I started doing that about two years ago basically with the directive to try to

help make WordPress better WordPress is all volunteer powered but there are some aspects of a project when it reaches that size that need really consistent attention so companies like GoDaddy that hire people to do that really help us keep things moving along I've been leading the team for a year and a half been on it since we started it I guess about eight years ago I've been contributing to WordPress for about 12 years but the story of how I got here starts even a little bit earlier than that who remembers this game we got like one person a couple people this is guerrillas it was distributed in 1991 with ms-dos 5 I was only nine years old

but I loved this game I played it constantly which when you look at it it's kind of strange because the graphics are terrible and the gameplay was equally terrible and this was the same time that the Super Nintendo came out and I had one of those and it sort of put this to shame but the thing that kept bringing me back to this game over and over was this gorillas was not distributed to showcase what pcs were capable of around graphics and gameplay it was distributed to showcase qbasic Microsoft's integrated development environment for the basic programming language so it was distributed as source code it was open source it may not be the kind of open

source that I've sort of grown to love and advocate for now because that first line right up there is a copyright but it really I credit it with shifting sort of sort of my focus from loving electronics and computers and just enjoying them to really being interested in how they work how I can control them and how I can make them do what I wanted them to do and this along with a couple other open-source things that nine-year-old me was able to get his hands on or how I started to learn to write code from that point of learning to write code it was about a year maybe as long as two before I started into

blackhat hacking breaking into systems back then was drastically easier than it is now security was not really where it ought to have been in the early 90s and it was partially because there was we were right at the beginning of kind of this shift where companies were even caring at all about security we were having sort of yes we were we were having this this changeover from sort of if you had the disks you had the program to things like activation keys that's sort of where I settled in but those activation keys the algorithms behind them were pretty terrible and it's because everybody was focused not on the technical aspects of their security they were focused on the secrecy of their

security and that is something that I think has been tough for our industry through those years at least from where I sit where lots of people have focused not on actual security but on secrecy as security if if they don't know how we're securing a thing then they can't exploit it and this is something that I try to push real hard against where I try to open up what we're doing as a project the struggles that we've hit the lessons that we've learned the tools that we use I love to get the chance to sort of share that with everybody when we started our team about eight years ago our goal seemed really awesome is just

keep WordPress secure right obvious except almost immediately we realized this didn't line up with our overall project goals which sounds strange maybe but the truth is from the very beginning of WordPress its focus its goal has been to democratize publishing to give everyone a voice not to make the greatest piece of software not to have the the most amazing code but to give people that tools so that they have a voice it was our users that needed to be the most important thing and so we had to shift from trying to secure our product which was already pretty big and already relatively old and so there were some challenges there but we had to shift our focus to keeping WordPress

users secure instead and it's important to understand sort of this mindset shift because it really affects all the things that we as a project including the security team do and and you won't really be able to understand why we're doing some of the things we do unless you realize that securing the WordPress code is only one piece of our overall goal and this was really the first lesson I guess that we learned if you will users over software our users are more important than our software itself this makes some things a little bit more challenging for example a lot of us that distribute products probably deal with this users that are using an older version are less secure

when your focus is on securing your software the solution to this is you put out a secure version of your software that was your job you fixed the problem their job is to use that right it's that's their problem the users problem but for us putting out a secure version was only the first step it didn't actually accomplish our goal until it was being used by the users and so we had this gap that we had to try to solve how do we actually get it all the way there in order to accomplish our goal of keeping users secure well for us that was going the extra step in upgrading for them which is how we ended up with

automatic updates automatic updates went into WordPress 3.7 which is like four and a half years ago so we've been operating under this mindset for quite a while and building automatic updates was hard and doing them securely and not breaking sites especially with all the different ways that WordPress is used it was a huge challenge but it was the only way that we could see to actually accomplish our goal of securing our users instead of just our piece of software it made a couple small things easier on us when we focused on users when we had that clarity we used to struggle a lot with figuring out when we should back port security fixes there was cons

discussion about how many people were affected and how big of a deal it really was and how hard it was going to be to back port it to previous releases and there was a lot of discussion and often a lot of of differing opinions but now our sort of decision-making flowchart for this is super simple does it help keep our users secure yes back port it and this is tough for us as a security team but it's a thing that we have embraced and continue to do all security vulnerabilities that are fixed in WordPress are back ported all the way back to either whenever they started or version 3.7 when auto updates were introduced the current version is 4.9 so

when we release a fix we are releasing 13 new versions simultaneously as well as patching the code that's going to be the next version it increased our workload but it helped us accomplish our goal we also realize though that even though this one little thing this one decision was maybe easier to make all in all securing users was crazy complex we had this code base that was under our control when our goal was just securing WordPress but when we had to secure users they all use plug-ins and themes that code is not under our control but does affect the security of our users so we have to care about it users host it thousands of different

hosts in at least as many different configurations and those are not under our control but it affects the security of our users so we have to care about it the other big thing that affects the security of our users is our users themselves who are not very good at keeping themselves secure as we look across the sort of I guess we get a pretty good picture of WordPress sites that are compromised by and large the biggest issue is still poor user practices usually around something as simple as passwords users are terrible at that so we realized if we want to protect our users we have to secure our software we have to figure out how to secure other

people's code we have to figure out how to secure their hosting and we have to educate them and all of those are problems that we have been struggling with and trying to tackle I'm going to talk some about the solutions that we came up with in just a little bit but from the very beginning when we thought yeah we need to educate users we got to teach them we know what they need to be doing we just got to tell them little bonus lesson here and educating users is terribly hard they do not listen well and for us we have tens of millions of users all over the world and they don't have to necessarily give us any kind of

contact info in order to download and use our software so we don't have direct lines of communication with them but it was important enough that it's an issue that we've struggled through and struggled through and eventually found some solutions to that I will talk about again in just a little bit to understand the next couple things you need a short quick I guess history lesson of WordPress we're old 15 years old is very old for software projects any software that's been around for 15 years and is still actively used and developed has changed dramatically and I think that a lot of our change has been driven heavily by our growth I don't have numbers going

all the way back to the beginning I don't think people cared to track numbers as much back then but in 2011 w3 Tech's started tracking estimated CMS usage percentage of the internet and the numbers are rough because tracking that kind of stuff is hard but in 2011 they estimated we powered about 13% of the Internet and the end of last month they estimated we powered about 30 1.4 percent of the Internet and that's a lot of growth and it looks very linear but I will tell you it did not feel that way when we were in it as you know working on the project that's because these are percentages of the internet and the Internet's grown

dramatically as well and if there's anything harder than tracking percentages of CMS usage it's probably counting the size of the internet but I took some likely terribly inaccurate numbers and plugged them in anyway because I think that it gives you a better picture maybe the numbers are wrong but the shape of the graph I think better shows what it felt like to go through this growth period and the years are labeled and that's pretty obvious but there's a few points where WordPress had to grow up a lot and I labeled a few of them in mid-2013 was when we finally got an actual build process for WordPress it was a little behind the times when we finally rolled that out

but it was really hard to do at that time w3 Tech's estimated we were powering 19% of the Internet and we had built a lot of tooling around our repository as it sat and that all needed to be redone because our repository needed to be reorganized it was a really difficult task because it was past due like it was way past when it was needed in late 2014 we finally moved to a reasonable form of communication we use slack now all the way up until then our main form of communication was a channel on IRC on you know freenode IRC that is how we ran WordPress when we were powering roughly 22% of the Internet and

then last year is when I think our security team really made one of those big growth spurts and that was when we launched our bug bounty program on hacker 1 and as part of that process we redid a lot of our processes procedures tooling etc but again it was it was really late in coming we were supposedly powering 28% of the Internet and we were running all of our security reporting for a project the size of WordPress through a security at email address that distributed to a handful of mailboxes of people who were supposed to handle that and the biggest thing that I realized through each of these really difficult growth periods is that we really should have been

assessing our needs far more often it's real easy to sort of get set up on what it is that you need for us eight years ago when we launched the security team that security at email address and our our couple tools that we used was great it was just what we needed but two years later we probably needed something else but we never really took the time to sit down and reassess our needs until we sort of hit a wall and it was impossible to do what we were supposed to be doing with the way things were currently set up and that is way too late to be reassessing growing iteratively looking at where you're at

every six months or a year if you haven't done that if you haven't sat down and said hey our processes are our tools right for us they were but are they still do so because these huge growth spurts they're not healthy and they're really difficult to get through I think it was maybe even a little bit more difficult for us in some ways because we're an all-volunteer security team that in general is very hard to do on any project and one the size of WordPress especially and about a year and a half ago as we were sort of restructuring and struggling through a lot of that I really was asking myself is it even possible to run security for

a project like WordPress with an all-volunteer team because volunteers are hard now there's some benefits to volunteers I think and I think it's important to point that out people volunteer to do a thing because they think it's really important or because they enjoy it and either of those things brings with it some level of passion and that path passion can drive performance and that's that's awesome I think it also brings us a diversity of input that we couldn't otherwise get our team is a little over 30 people and some of them worked for hosts and some of them work for security companies and some of them quite a few of them worked for various development agencies some of those

agencies focus on enterprise some are focused on app development with WordPress as a back-end all these different ways that WordPress is being used and our team is actually doing that so we have all that variety of input and I think that's really healthy but there's some massive downsides to a volunteer team and the biggest one is that available hours fluctuate dramatically right people give you you know they're dedicating a few hours or a handful of hours to volunteer but when they have a big project at work or they're coming up on a deadline or they have some big life event going on they just don't have hours to volunteer it's not that they don't want to they simply

don't have it available and the number of times that that seems to happen to a large percentage of the team all at once is a little bit mind-boggling so we had to sit down and try to figure out how do we make this work because all of WordPress is volunteer powered like this has felt really normal all along but the security team needs some consistency we really do in the past the solution had always been grow the team if we needed the number of hours that we were getting from five people but those people were busy you could grow the team to 10 people and now as long as only five people were busy the other five could

handle it that had been our solution to this all along in the past but when so that was my first go-to when we started running into this a year and a half ago but it didn't really work this time I think that we had hit a scale where simply growing the team was just no longer the solution the team got so big that it wasn't that people didn't have hours to volunteer it's just that they weren't and it was mostly because everyone said there's so many of us somebody else is going to get it right somebody else is going to take care of it and it completely backfired did not work at all so we shrunk the team back down and the

thing that we did successfully land on is that we now operate a little bit as a hybrid team we're still mostly volunteer the vast majority of ours are volunteer hours but for example I'm paid by GoDaddy to work on WordPress and a lot of those hours or sank into WordPress security similarly there's a handful of people on our team that are given a certain number of hours by their company paid usually it's like a 20% project so they get like 8 hours a week and a couple people put that all into security and a couple people sort of split that back and forth between core development and security depending on where they're needed and this has given us the ability

to sort of reap the benefits of a volunteer team but also never completely have no hours available because we need that consistency our tools are completely focused on making us more efficient making it so that we can do more in less hours because ours are our most precious resource with volunteer team we've tried lots of really new great fantastic tools and what we've settled in on is actually a really simplistic set of tools we use slack as our main form of communication I mentioned that we have a private channel for security we've also got some scripts set up that let us let anyone on the team easily spin up a security sub channel that automatically invites the whole security

team and allows us to ad hoc invite certain people that may be helpful in discussing a specific issue but don't necessarily need to see everything that's going on we have a couple of those channels that are actually permanently spun up that I'll get to in a little bit that have very specific purposes we use track some people love it some people hate it mostly the latter but the WordPress project has used track since the very beginning and because most of us have been around it for a while we're really comfortable with it and so while we've tried a lot of much better bug trackers this one seems to work the best for us and so we have come

back to it it's a completely internal facing tool so it's only something that our team needs to use anyway and we use this mostly for putting up patches on things doing code review on those iterating on them etc etc and then we use hacker 1:yes to power our bug bounty program but also as our main sort of way of communicating with reporters and researchers and keeping them in the loop and in this big changeover where we sort of updated our toolset and settled in on this and tried a lot of other things the biggest thing that really stood out to me as we test it and got rid of a whole bunch of tools is that tools don't fix

most of our problems they often help highlight where some of those problems really are we think that they're because of our tools but for us most of them weren't for example we really struggled when we used a security at email address to keep hackers in the loop as they reported a thing to us and we focused in on on working on it we often went radio silent didn't really keep them them updated and they mistook that radio silence as nothing happening which is kind of understandable and it's strained relationships we had a tough time keeping people updated and now we thought as we sort of piloted the hacker one tool and we saw how amazing that was

for keeping people in the loop how easy it was no longer were we staring at an email going somebody else is probably going to reply to this and then finally saying okay fine no one's replying to it I'll reply to it and then it you know six replies all came through at once there are all these downsides to this old tool so we thought this new tool it's gonna it's going to fix the issue for us and then we went through our whole test we we launched on hacker one and they're still terrible at keeping researchers in the loop and it's because the tool wasn't the problem and so the tool didn't fix it the way we fixed that

particular problem is adjusting things around so that we no longer assign issues to a single person we have these small like three to four person teams the issues get assigned to so that there's a handful of people they can see that someone needs an update but also in just kind of bringing in some new people to our team who are focused on keeping communication flowing well right we had people who were good at fixing issues we had people that had historical knowledge of our codebase all this sort of classic stuff that you expect a security team to have but we needed those communication people that wouldn't get distracted by fixing the issue and would be focused in

on this thing that we were not doing well so making those adjustments fix the actual problem when we thought tools were gonna fix it which brings me to one of the sort of biggest lessons that we've learned over the last year and a half two years we have struggled for quite a few years with how do we take our security all the way to the users with these these three big areas that seemed like problems that that were almost impossible to solve write code that wasn't under our control how are we going to how are we going to keep users safe when they're using code that's not under our control hosting environments that are also not under our control and

educating our users that we tried so many different ways and none of them really seemed effective and the crazy thing is these three biggest problems that we had been struggling with as a team for four or five years now at least were actually all solved with the same basic thing which was building relationships I think that we often at least the WordPress team definitely did we often overlooked the fact that there are other people that can help us accomplish our goals no matter what those those goals are and so we took the time to sit down we said okay we're having these struggles if we controlled everything if we controlled absolutely everything what would we do well we

would make plug in code more secure so we need to start building relationships with the people that do control that one of our permanent sub channels in our slack is for communicating with plug-in authors right now it's just our security team and the authors of some of the biggest plugins in the WordPress space the ones that are on maybe a million or a few million sites but we've been working real heavily with them to try to make sure that their security works well with ours that we're functioning well together and that we're setting a good example at the top because in our ecosystem everybody that's building all these other plugins tens of thousands of plugins are kind of looking to the top

for either examples or to straight-up copy code and use we've built a lot of relationships with hosts laughs CD ends trying to see where we can protect our users in ways that we were never able to before right WordPress could only when we were just securing our own code we could only protect users at that application layer once they got all the way down to WordPress and by building these relationships we've now been able to when we see a problem that could be fixed at the network layer or mitigated at the network layer we can get all the major hosts and companies like CloudFlare and such to start protecting millions of WordPress sites before we

even push out any fix and for educating users all of these same companies that we're building these relationships with have much more direct hands-on relationships with our users so we've been able to work with them to help build some education programs and help them to educate our users building better programs through through hosts building even some things through search engines like Google and their their search engine console being able to work better with them to use them as a channel to educate our users I wanted to leave a lot of time for Q&A because when I throw out talks like this it's always interesting to see what kinds of things people are interested to know and

we're really open as a project and would love to share whatever it is that you're interested in that you'd like to know oh and there's a mic to run around I almost forgot because we are live streaming fate I have a very small CMS that's not open-source but I'm thinking about open source but I'm kind of terrified because then everyone would know where the holes are and you know I think I'm smart but I know that I haven't caught all the holes is there a recommended path or some RTFM you have for folks who are thinking about going down this route so that's that's really interesting we get that sort of mentality a lot people say well

it can't be secure because it's open source and everybody can look into it and find the security vulnerabilities and obviously there's some danger that someone's going to see a thing that you didn't see they're gonna find that vulnerability and they're going to exploit it it's possible I mean you're giving them a peek into your code but similarly they may find that vulnerability and want to exploit it but somebody else may find that vulnerability and let you know about it I think that if you're going to go the route of open sourcing a thing it can actually be really beneficial so many eyes on a thing help find all these issues that you may not have seen but

only if you have good methods for them to then report it to you and ultimately get it fixed I think that that's the biggest thing if you're going to open source I don't think that makes you inherently less secure but make sure that those channels of communication are in place so that people can help you keep your products secure rather than exploit it in some way that they find and in general I think that this has been great in the WordPress project as far as we have we as a security team yes we find issues but tons of other people have found issues over these 15 years as well that we've been able to fix and and

when we fix it because they noticed it it's fixed for everybody and that's that's a big plus thanks actually wanted to comment about the lost gentleman's question there you know in the WordPress community it seems like we went through this phase in 2011 2012 2013 where a lot of vulnerabilities came to light and I think what might have happened is WordPress hit a level of popularity where it became very interesting to security researchers and so what you might find is at the stage of growth that you're at now it's not super interesting for people to do kind of pro bono research into vulnerabilities in your platform but you also don't have a big user base so

they're not going to be a lot of people affected by potential vulnerability so I would say if you open source it now as you grow you know you're probably going to find that you hit that hopefully as you grow you're gonna hit that phase where now you're really interesting and on everyone's radar and you go through a bit of a challenging period and you have to scramble but but then you get past that and in in the WordPress space now we're in a much better place than we were in 2012 I would say Aaron yeah I think that since probably right around 2012 or so we have felt like we had a giant target on our back right like

because everybody was if you were trying to compromise websites WordPress was a great way to do it because there were so many WordPress websites out there and there is sort of that that rough period to get over where you're interesting to people but you don't necessarily have enough help from other people now that WordPress is run in a lot of enterprise sites you know Disney and and Oprah and some of these big companies that actually do a lot of vulnerability testing and research and stuff and bring it back to us and help us with that sort of what I feel like got us over that hump and there are those awkward growth periods that I think are just a little bit more

awkward in open-source you kind of got to prepare for those yeah right [Music] yep the the very the very beginning of just putting it out there and somebody seeing it it can be scary I've released quite a few open-source things and it can be scary but ultimately I've found that if they continue to grow that it's been ultimately beneficial for the projects that I've done that way so there's a lot of time of course then on like responding to and fixing security vulnerabilities that either you find or other people find I'm curious what your thoughts are on like the other side of that of preventing new vulnerabilities from being written especially when it is like a lot of volunteer contributors and

like kind of maintaining standards of code quality including security yes so as a security team we do a lot of code review obviously a lot of people are interested in writing new features and adding new stuff and that's cool and fun and neat and shiny but checking to make sure that what's being added is secure there's a much smaller group of people that consider is that fun and even then that's only some small percentage of our team right but we understand it's important so we do spend a lot of time doing that kind of code review especially around big new things that are being added into WordPress right now a complete reworking of the way the

editor works in WordPress a whole new system for that is coming in it's a ton of code that we've had to continually review and look at some of the ways that we're able to sort of keep up is that we've built some tools to keep things simple things like code quality and code standards are are checked with every commit while anyone can volunteer to you know put in some some work and write some code and get some patches up our committers the people that can actually put that into the project they you know they have a little bit better knowledge and can are kind of the first line there checking that but yes we spend a huge amount of

time just making sure that what's already you know secure as secure as we can make it doesn't slip back into some major problem well my question actually can end up an extension of that one but assuming you've run into friction when dealing with some feature development teams in terms of them saying oh no go a security we don't want to listen to you what are some of the strategies that you've adopted to kind of get it to become a win-win and get them on board and get their buy-in and have you kind of shifted along the one longer journey in terms of like we're just gonna force this on you whether or not you like it

or not versus a more collaborative approach so some of the things that first spring to mind are that we've had some struggles in the past with when when these new features want to make use of other libraries that already exist as a security team especially as the lead that kind of handles a lot of those relationships with other projects I feel like we not only have to review that code but we need to communicate with the authors of those libraries that we're going to use and make sure that they are going to work with us when issues are found right all these kinds of things and that's been a struggle less that people who want to use these these

libraries or whatever want us to go away but more that they just start their building they're doing a thing they don't even loop us in you know we have to really pay close attention and notice it ourselves so that's been one of the struggles is just being aware enough of that aside from that probably the biggest actual friction that we have is this constant balance that needs to be found between securing all the things and keeping a product that's easy to use and flexible right the the thing that I think has helped with that actually is really working with our security team to basically say look I know that we all want to secure things as much as possible but if our

goal is to make our users as secure as we can make them then if we make it fantastic but they don't use it that way it doesn't actually improve their security so we have to find this balance point which is a real mindset change but we have to find this balance point where we're doing as much security as we can that is bringing those users along with us and sort of maybe iteratively grow that and so working with our security team to say okay sometimes it's not going to be absolutely perfect because we have to work with in reality and then yes having to get that buy-in from the teams that are like no no this is so

much easier yes but we have to fight we're gonna give a little but you've got to give a little too we need to find this balance point in the middle yeah yeah and work on another large open source project and we're kind of at the stage where we have a product security - lock channel and a build process but not a bug bounty yet and there's a lot of discussion as to having a bug bounty I wanted to hear a bit more how you guys set that up from a point of like scoping like we're talking about to do plugins get included or not if they're not in core and the team the scree team is really

afraid of opening this bug mone and just being like everybody's gonna be busy for six months like so how do you kind of manage that that type of expectation so I'll tell you how we manage it now in just a minute but it's we didn't manage it enough when we first launched and that initial if you're a pretty popular open source project that initial sort of hit that you get when you first launch that program is pretty intense we were very overloaded for a period of time we should have run longer we ran a sort of a private test that was invite-only we use hacker one we found that that's been a fantastic tool for us but we don't

include plugins we only include all of our products and projects so we have WordPress as well as buddypress BB press and WP CLI other things that we build and all of our sites are also under that so our wordpress.org site all of our event sites all that and some of the things that we should have had more in place is extreme clarity you probably have some of those issues that are reported relatively regularly and aren't valid not just real clearly specifying that on the page but also have some sort of are you sure kind of trigger system that warns people because it's not for us it's not the the number of valid issues that are being found that's the

struggle and it's not the monetary expense that's the struggle it's the noise that's the struggle it's the number of invalid reports that you sort of have to wade through to get to the valid ones and I think that that's partially just a volume thing but also it's not fun to wade through in valid reports and a volunteer team who wants to volunteer to do a thing that's not fun right so with hacker one we helped pilot a program that they put out called human augmented signal where they have a team that's kind of the first line of defense to filter out the most obviously invalid reports that come through and that has been absolutely amazing for us in the

last year of our sort of the well in the first year only about 9% of reported issues were actually valid but of the of the ones that were invalid more than 50% of those invalid reports were caught by that hacker one team and never made it to our team which is pretty great so looking at things like that that might help you through that initial sort of skyrocket in number of issues reported can be really helpful so your second lesson was reassess regularly and what I think is kind of interesting about that as I'm sure your entire team looked at all the stuff that they had on their plate and it was never important for any given build process security

process or whatever to be addressed until crisis hit and I've been at companies that will remain nameless that have let the crisis point come and go and then still haven't built the changes and now are facing a real hard problem so the question I have to you is the is there anything in hindsight that should have been a warning signal that you are hitting the crisis point before the crisis point became blatantly obvious to everybody because everybody on your team was smart right and everybody was saying well yeah somebody's complaining that they need better tools but we've got something more important to work on so I think that I don't know the like those those things that may tip you off

very dramatically right depending on on what kind of crisis you're about to hit probably but for for us I think that one of the reasons we missed a lot of those wasn't because the team wasn't smart enough to figure them out or recognize them it was honestly because we didn't have a good hierarchy within that team and so there wasn't a a person kind of focused on watching for those things we really didn't have a security team lead we just had a security team until I think it was three years ago that we we sort of said hey we should have somebody in charge and at that time it wasn't even in charge so much as it was a

person to to put their name there that people could go to if they if they wanted to talk to building in some hierarchy where things were flat before sets some expectations that the people at the top are watching for those kinds of oncoming things that's the thing that I take on myself for the security team and similarly in the WordPress project as a whole we only ever had release leads someone that was in charge of the next release and then somebody else would be in charge of the next release I led WordPress 3.6 that was my thing I did that but I didn't also lead 37 38 39 like I didn't have some long term plan

that everyone was bought into so getting some of that put back in place I think has helped us as well to see those kinds of things coming speaking of crisis with WordPress being a platform has created or developed and there's a largely volunteer team how do you deal with Incident Response when there's a huge announcement of vulnerabilities something came to mind was like the Panama papers that that was largely a wordpress part of that was largely a wordpress plug-in vulnerability and it got a lot of attention is there like an incident response protocol that you follow as a team that's a thing that we're in the process of still trying to build and figure out

right now are our teams a little bit [Music] just sort of let's jump to it and all hands on deck and let's figure this out and we're sort of assessing each one as it comes in and figuring it out for that issue which I don't think is healthy so we've gone through a lot of these changes and fixed a lot of the other things that we were dealing with and that's hopefully that will be one of the lessons that I've learned by next year and maybe I can come back and really explain how we changed to figure that out but right now we don't have like a response that we follow every time we're sort of custom building those solutions

each time it's so kind of building a little bit on that question and an earlier one multi-party response wondering if you have any lessons learned or experiences to share about common vulnerabilities that affect multiple companies and how you respond to those for us one of the I mean again a lot of this comes down to relationships right there's um there's been this long-standing thing where for example every CMS kind of does its own thing WordPress hasn't really worked with for example like even other open source ones Drupal and Joomla and stuff in our space we had hardly ever interacted with them on on these kinds of levels even though a lot of security vulnerabilities really

do often affect multiple platforms we now work pretty consistently with the security teams from these other CMS's so that when those things happen we can do things like coordinated releasing and that kind of stuff most of that actually seems to happen on our WordPress slack we spin up channels and bring those people in the security team leads from those other projects are on our slack already anyway because we've been trying to communicate regularly so that's that's where we tend to handle that but yeah that's a thing that I feel like we've gotten better at over the last couple of years anyone else well I'll be around for a while if anybody else has any other questions

come find me and ask thank you [Applause]