Brian Vallelunga
Appearances
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Agreed. I do think that the world is changing. It's just going to be very, very slow. So like you can see this in passwords today where we're going from passwords to pass keys. So anyone familiar with pass keys, it's basically like you can sign into a website with like your face or touch ID and you never had to put in a password. It's a passwordless world.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And the way that's working is through cryptography. You have like a fingerprint that leads to a cryptographic signature that can be used to authenticate yourself. And that that thing that's happening in passwords is now starting to happen in secrets with identity based authentication.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so like the kind of picture that we can start to see happening very, very slowly in like Doppler's job is to like help move this along, help move the industry along is to have like this bridge of connection. So like on one side, you have like your AWS GCP execution environment, and that has an identity already today. And then you can pair that with Doppler.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And now you're authentic by the fact that you're in that EC2 instance or you are now automatically authenticated to Doppler. And then eventually Doppler will have the connection of now that you're authenticated with Doppler, you're automatically authenticated to Stripe. There's no more API key needed.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
We are all cryptographically authorizing each other across this chain or this highway of authentication. And that's kind of like the dream state that we're going to be getting to over the next couple of years. It's going to take a lot of work of working with partners, but you don't have to convince Stripe, Twilio and so many other providers to enable this.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
You should kind of treat your environment variables and your secrets all the same, right? Your port and feature flag should be treated as securely as your secrets. That way, developers don't need to make the choice of is this secure or is this not? They're always just doing the secure thing by default.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And we're going to try to make as easy as possible, but it's going to be it's going to be an uphill battle. But if we can do that, we're in a secret.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
There is a little bit of an open standards, mostly between like AWS and X like Doppler or whatever it may be, but there is no open standard on, or even a standard altogether on the like secrets manager to third-party service like Stripe or Twilio. So I think our goal is we're gonna probably try to create an open standard
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
that both sides adhere to so that if you're anywhere in this chain, you can directly, you can just plug in very quickly. And I think we'll probably have to build SDKs for like every primary language and Kubernetes and a bunch of other infrastructure tooling as well. So just like works out of the box.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
All right. So I have four questions that I think every person listening should ask themselves. And it's like basically a very simple sniff test to see if I have a problem. So to highlight what the problem is, it's called secret sprawl. And that is basically all your secrets are scattered across a bunch of different places.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And because of that, you have no way of controlling it or governing it, which is like one of the most important parts about protecting your secrets is being able to govern those secrets. And so the four questions are, can you confidently answer where all my secrets are?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Not just the ones that you know about in a secrets manager somewhere, but all the ones on developers' laptops, all the ones in Slack, in email, in your code somewhere. You got to have a complete picture of where all your secrets are. And you got to be able to make that guarantee that you're not going to miss one by not knowing about it. And I'll talk about how to solve that in a second.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
The second is, do you know who has access to all of those secrets? And this gets really scary, especially if you don't have like a centralized system, because like if I have a file, I can just share it to another person and no one on the security team is going to know about that. Can I control who has accessed those secrets when they were accessed and by who? Right.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And then the trick is just make that that whole workflow and tooling as developer friendly as possible. So they wanted to go down that path.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So do I have access controls and do I have an autolog associated with those access events? And then most importantly, when a data breach does happen, can I stop it very, very quickly? When I say quickly, I mean in the orders of seconds and minutes, not in the order of days and weeks or months.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so if you cannot answer those four questions confidently, like to the point where your job would be on the line for it because it essentially is on the line for it, then you have a problem. And I can talk about in a sec what it looks like to have a solution there.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
I don't have to go into like Doppler's features, but I can talk about like the fundamental properties you'd want to look for in any secrets manager, Doppler or someone else.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yep. And I think once your surface area grows of team members, infrastructure, Devices, it gets pretty hard to answer those questions confidently, unless you have a dedicated system for it. So I'll be valuable to go into like, what actually does a dedicated solution look like? The first thing is you can centralize all your secrets in it.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So once you know where all your secrets are, you can confidently say, this is the place we're doing. We're not gonna store our secrets in code. We're not gonna send it over email or Slack or Microsoft Teams or wherever it may be. This is the one place you can get all your secrets. Now you have a complete landscape of your secrets.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
The next thing you should be able to do in this system is you should be able to can set up access controls and audit logs around them. So you can say you have this develop full stack developers have access to the development environment for these sets of projects for the teams they're on. But the DevOps team has access to production.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So you can start getting access and you have a full audit picture of it. And obviously that audit picture then gets pushed into your auditing tools like Datadog, Simulogic, Amazon, wherever maybe. You should be able to have this infrastructure natively integrated with their developer workflows for local development.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
That way developers aren't working around the system and then recreating that risk you're trying to avoid. And it should be natively integrated with your production and staging infrastructure so that When it is time to roll out a secret, you don't have this race condition of the code arriving before the secrets do. The secrets have to arrive first if the code is going to rely on those secrets.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so then this kind of gives you this whole picture of I know that if I can see the secret in this project, in this environment, on the solution like the secrets manager, then it's guaranteed in my production infrastructure or it's guaranteed on my developers laptops in a secure way. And that basically gives you the ability to remediate secrets pretty quickly.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so the last part of this puzzle is being able to rotate secrets. Rotation is the fancy way of saying, I'm going to swap out the locks on the door. And you got to do it in a way because if you just like immediately destroy the lock, in this case, you'll just bring yourself down in production. So you have to do it in a way that does not create any downtime.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And it's usually requiring a two key system. So like in a database, for example, you don't ever want to not be able to connect to the database. So you'd have two credentials to the database and you swap them and then you rotate one and then you swap again. And so you want a system that does all that. That should not be done by humans.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
That should be done by a system that's running on a schedule where you have an immediate like I'm fucked button. I need to click this thing to get me out of a data breach. And then obviously it orchestrates to the rest of your infrastructure. And so those are the things that you should be looking for in a developer for secrets manager.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
A good way that I kind of like talk about it when I'm talking internally with our employees is the game of offense and defense are very different games. The game of defense is build walls around your perimeter as high as possible, as strong as possible, with no cracks in the foundation, right?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And then the goal of offense of an attacker, of a hacker, is to find one crack in the foundation and hit it as hard as freaking possible that brings down the entire wall, right? They only need one weak link, right? And if secrets happen to be that weak link, well, then you're really giving them the keys to your digital kingdom. Um, that's extra bad.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
I'd much rather be the weak link, be something far outer in the perimeter. Like, um, I don't know, some authentication bug or something like that, that, that impacts like one or two users, not your entire users, uh, user base. Um, And so, yeah, managing risk is incredibly important here.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
That's the end goal that you're trying to accomplish is, like, make sure you can stay up and orchestrate, like, get all the DevOps and platform tooling you're looking for, and then also manage the risk associated with that.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
We have tried really hard to build a model that maps very closely to what you'd have with code, so with GitHub or GitLab. So we have projects just like you have repos in GitHub and GitLab. And we have environments which are very similar to branches. And then we have user groups that map in with access control policies. And then we have integrations that connect to those environments.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so you can have these like this bare bones, hierarchical structure of projects and environments. From there, you can assign users to them and you can assign infrastructure that connects with them. And it's very easy within Doppler to kind of see all that connect together in an extremely visual way. We even have to show kind of how far we've taken it.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
When you go into a project, you actually have a Kanban board style user experience that shows all your different environments from development to staging to production, everything in between.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah. It had nothing to do with secrets when it started. I was working on a crypto machine learning marketplace. All the buzzwords in one. It was literally every buzzword you could find jammed together. And man, that was a hard project. I worked on it for about eight months straight and it felt like pushing a massive boulder up a hill.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Every foot forward, I'd slip five to six feet back from exhaustion. It was just hard. And at one point I really got to my breaking point where I was like, I don't know if I can keep working on this thing. And so I decided to take a trip to Cancun.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
mexico and the whole goal was like don't think about this project um actively do not and obviously i failed at that miserably uh i was thinking about it the entire time i was there and what i realized is that this pro i was never going to be able to get this project off the ground but that doesn't mean that i had to stop there um i think like there's a lot of people that like like follow like sports players and like they know all the stats of all these players i'm very much the same way with founders i i like i love learning about founder stories um
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And I think Stuart Butterfield from Slack is like the best at failing upwards I've seen. Creates a video game out of that video game. The video game fails. Born out of that was Flickr. Comes back a second time. So I'm going to create a video game again. Born out of that was Slack, which we all use today. And so I was like, what can I learn?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
from um from my experience with this crypto machine learning marketplace and managing secrets was just one of them that was like super painful in all these weird interesting ways um it was weird from a developer productivity side because i always felt like i was copying and pasting things in but sometimes the code would get into production faster than the secret force at an outage um i had a couple of developers that i was working with on this project i was like really painful to sync all these secrets with them like i would literally slack them and email them
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
the secrets and I like my heart would cringe every time I do that. Like, oh, that didn't feel right. But that's the only thing I have at my disposal to do. And there was actually one time where we dropped the crypto part of the marketplace because it wasn't working. We just did straight up machine learning marketplace using Stripe.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And I had the Stripe production key in staging and the staging one in prod. And it took us a month to figure out why we weren't doing any transactions. And it was because of that, because the Stripe testing key rejects all live credit cards. It only uses test credit cards. And we didn't realize that for a long time. And so I was like, okay, this is a real problem.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so I come back from Cancun and I go to this dinner that Stripe was hosting that had a bunch of founders and developers at it. And we were all going around the room talking about what we were thinking about. And I just asked them openly, am I a shitty developer? Is the world broken? I just can't tell anymore. Are you guys having this problem too?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And like 60% of them were like, yes, this is a very big problem for us. And one woman in particular, like literally comes running up to me right afterwards. I thought she was going to trample me. And she says, I've had three outages this week. Have me a solution by Sunday. And I was like, no, no, no, no, no. Like I'm not even started yet. And this is a Wednesday. There's no way that's happening.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
She goes, I don't give a fuck. Have it by Sunday. And I was like, okay, there are some interests here. People want it. Um, But we didn't obviously have it by Sunday. That never happened. But what we did is we started doing a lot of research and we found that individual developers are struggling because there wasn't tools made for developers.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Medium sized companies are struggling building their own things and large companies were buying hash core vaults, which really isn't meant to solve this problem and still struggling. So like everyone's struggling. And I was like, okay, we can build something that's developer first instead of just purely security first. And we got our first version working in three weeks.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And we got a set of customers right after that by just dragging them to Chipotle and saying, hey, I'll get you anything on the menu as long as I get a rant at you for about two hours. And that got us our first set of customers and off to the races we were.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah, it's a that's pretty much the two paths that we see, either some encrypted file somewhere or it's just directly available in the environment. I'm a pretty big fan of. if you're going to do the environment path or inject it through the environment, immediately do a cleanup of the environment before all your other dependencies could get loaded in.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah. Anyway, but turned out to be a good business.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah, I would probably index and be like, okay, are other people that I know that also do podcast recording asking for this tool? Does it continuously get better and better the more I mount onto this hodgepodge of a product? And is that value going up exponentially, not linearly?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And if both those answers are correct or yes, and you still feel like there's nothing in the market that does anything close, you have a product. The question is, is that product 10X better than whatever's in the market? Because if it isn't 10X better or 10X different, there won't be enough of a differentiator to create a buying event, I found. Right.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah, there's a couple that really come to mind. I mean, let's look at... He laughed like he's got some doozies of some stories. They're all public. I'll share one that isn't public, but I won't say the name of the company, and then I'll share a couple that are public. So Twitch, Toyota, and Twitter all kind of had the exact same breach happen. And they had secrets in code.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
The code got leaked somehow, either by them opening up the repos themselves, or by hackers being able to get access to the GitHub account from an employee that got compromised. So like someone... got called and said, hey, I'm your boss and give me access to GitHub. And then the hacker got access to GitHub. And then from GitHub, they saw the code and the code had the secrets in it.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And boom, now the hackers are really off to the races. And that's like that happened multiple times and keeps happening because there wasn't a system in place to make sure secrets never entered code. They should never be in code.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Absolutely. I mean, that is like the number one thing most attackers are going after is like GitHub because they're assuming you're going to be making this exact mistake. And it's very easy to impersonate a boss, especially in the days of AI. So these attacks are going to get more and more sophisticated and feel more and more real.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Like you're actually talking with your boss and all of a sudden, boom, you have access or they have access. And then I'll tell a story. I can't say the name of the company. I can promise you this. You have absolutely heard of the name of the company. You probably have used their product at some point in time. And they're very big. And this is before they were customer of ours.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
They had a data breach where they hired a malicious actor in the company. So there was a rogue agent that they had hired. That rogue agent stole all their credentials or all their secrets, about 2000 of them. They detected a bit late in the game, obviously exited that employee. But then the security team was tasked with rotating these secrets.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So like basically load those secrets out of the environment, then clean them up from the environment, then load everything else. That way you can't have like a rogue dependency, go and capture all your secrets and then send it somewhere. The other path of like using a file that's encrypted is also a really good path too. So I think both are pretty good.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So again, swapping out the locks, making sure that all those secrets they stole aren't valid keys to the locks anymore. And it took their security team six months across three engineers to rotate all of these credentials. So six months.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
That's not just like six months of only doing this and not doing any other projects and a major distraction, but also six months that the attackers got to steal data. Well, if data moves at gigabytes per second through AWS, They have an enormous amount of time to drain all the data, right? Like they had all the time in the world, basically.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Like when you're dealing in data breaches, you're dealing in seconds and minutes, not months. So months is an absolute insane amount of time that the attackers had to basically use all those stolen credentials to get anything they wanted out of the system or compromise the system in any way.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Um, and it was from what I, what we heard, it was quite a bit of a battle between the hackers and like, just because you wrote the key. Well, if they use that key to grant, to create new keys, now you're like, you have this cat and mouse game. Right. Um, and so it was quite a big challenge for them.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Uh, post Doppler, they, uh, have a secret station set up a whole level of orchestration with their infrastructure. And that same thing would have taken about like 30 seconds to a minute. They'd clicked a button and it'd been done. But, yeah, it was quite painful for them. And you can imagine all the users that were deeply impacted by that along the way. It's not fun.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
I think that the surprising thing is like that I hear a lot is, oh, this won't happen to me because my company is really big and we had the best security team. Or I also hear the quite another inverse, which happens all the time of like, I'm a small startup. Hackers won't care about my company. Right.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And like you hear this from both from the opposite sides, like the stars are saying one thing that the big companies are saying the other. And the truth is hackers are just looking for any company that they can exploit. Does not matter big and small. They're looking for prices. Sometimes they hit a jackpot million dollar payout. Sometimes they get a 10K payout. Right.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And like our small a small attack or a big attack. But they're just looking to make an exploit. And so really every and that's why the number of data breaches are just rising exponentially every single year. Like if you look at the graph, it's terrifying and it's getting and it's getting cheaper and cheaper for them to do that because the techniques are getting more and more understood.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
The AI is getting better and better at doing this. So it's a really big growing problem. So everyone is vulnerable. It's not just if you're a startup, you're insulated because of your scale. Or if you're a large company, you're invulnerable because of your scale. Like Microsoft got hacked. Nvidia got hacked. Samsung got hacked.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Like these are really, really large companies that you'd think would have their stuff together. And still, they got hacked. So it's on all of us. All of us are responsible for keeping these credentials safe so we keep our users safe.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Agreed. Like when you're in a firefight, you shouldn't be strategizing how to fight the firefight. You should just be doing the strategizing. Should it happen before the firefight? Right. You just want to be in execution mode when that happens. You want the execution mode to be as quick as possible. Like you should click a button and it should be done 30 seconds later. And that's it.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Oh, yeah. Like there's some interesting ones that I've heard of recently of like printers, like old printers that are connected to Wi-Fi getting exploited or like everyone has all these IOT devices in their home that are connected to like the home hub system for Apple or Alexa or whatever. And those are internet connected too.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So like a very small device can then basically get access to the wifi network, which then gets access to every device on that network. And thus the attack starts. So there's a lot of exploits outside that don't have to start with just your computer or just AWS.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah. Um, well, first off Doppler is super cheap. We're free actually to start. Um, so it doesn't get cheaper than that. Um, uh, yeah, it's free for the first three years users. And then it's basically GitHub priced from there. Um, so if you, if you're comfortable paying for GitHub, you should be comfortable paying for Doppler. Um, uh,
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
But let's just say you don't want to do that for whatever reason. I think the most important thing you can do if you're not going to use any tooling at all or bare minimum tooling is use like a cloud secrets provider, like AWS Secrets Manager, GCP Secrets Manager, and so on, and then build some tooling around it so that you can fetch secrets locally in development without them touching disk.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So what you need to do is have some wrapper around your application that starts your application. So I'd imagine the workflow would be like fetch secrets from AWS Secrets Manager, inject them into the environment or write them to an encrypted file and then have the decryption key in the environment, either path works.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And then once the application starts, you clean up the environment and then load all your other dependencies. And if you did that, plus having your secrets in AWS Secrets Manager and that being the same process happening for production workloads, at least you've solved most of the risk, in my mind.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Also, when you're doing this and this where things get a little bit more of a burden because AWS managing privileges like IAM roles is quite complicated, in my opinion, is you're going to want to make sure that every individual developer and every developer set of infrastructure has their own identity.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And that way you have robust audit logs of like, okay, the secret was taken by X developer at this time or read by this time. And we, we can, we have this auditor of what happened. Um, so you don't want to basically create one master. I am credential and just share it with your entire team because then you lose a lot of your auditing story along with it. Um,
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And you'll also probably want to set up, if you have a little bit of extra time to spend, an auditing story around reads. So every time a secret is downloaded from this endpoint, you're going to want to make sure that you have the right logging and auditing set up in AWS for that. And that you're actually looking at those logs and have analysis over those logs. Gotcha. Absolute bare minimum.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Less and less. So they added something called one password secrets automation. Everything that I've heard from my community is that they are not actively investing in that product anymore. And it's kind of like Shelfware now. Shelfware, I've never heard that. It's there, it's available, you can use it, but it's not actively being invested in. It's on the shelf.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
But the amount of people every year that I hear using it is less and less because, A, I think there's first-class products now that really work. And the amount of things that you want out of Secrets Manager is growing far faster than what they're adding to, which is at this pace, from my understanding, it's zero. But you can't do it.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
I mean, like Secrets Manager or 1Password is end-to-end encrypted. So like you're getting a lot of security benefits from that. So I'm not going to say, hey, don't ever use it. If it push came to shove, use it. I just think there's like far better things that are also easier to use and far better priced out there that are readily available.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So are you I'm guessing you have your secrets stored on like AWS secrets manager and then those secrets are then pushed into the container.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yep. That's awesome. We are. If there is a Doppler token, a Doppler-issued token found on GitHub, GitHub will contact us immediately. We'll immediately revoke the token, notify you, and work with you to generate a new token. And we're going to be investing next year pretty aggressively in the secret scanning space as well. Like really having a full auto picture there.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Like I'd love to be able to say, hey, we found some secrets in your code. We've automatically revoked them, issued new ones. And those new secrets are now not in your code anymore. Like some really powerful workflows like that can just make sure that even though the tooling is set up so that you'd want to use it, if you actually don't use it and you fall back to bad habits, the system catches you.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Detection ahead of the breach, at least in the secret space, is almost zero from my understanding right now. There isn't a class of tooling yet for that. But on the scanning side, there's TruffleHawk, there's GitGuardian. GitHub is adding some stuff too. There is a space developing right now where basically you can scan. The goal is to scan the perimeter of your infrastructure, right?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So like Slack, Microsoft Teams, email, your code base.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
all of that infrastructure um basically will be scanned in real time for for secrets um that's basically the extent of what we have today the problem is like you have to connect those dots right so like you found a secret what do you do next right especially if you're like an individual developer you don't control the secrets manager that's being used you don't control
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Or if they're not using a secrets manager, what the flow is. And so like if you're in, if you like a classic example that I hear all the time is, oh, we added a secret scanner and the secret scanner found all of our ENV files. But the company has decided we use ENV files.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So like now it's a moot point because we know we've always known secrets were in code because they're in ENV files, but there's nothing to get us to the next step of like getting it out of code. So now all you're doing is surfacing a problem that we can't solve. So I think that's really where like we're trying to help move the industry is connecting the dots.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
You found a secret and you can immediately take an action or an action is taken on your behalf based on a policy to clean up the mess.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah, we have a couple blog posts that highlight basic secrets management practices. Outside of the stuff I already shared today around working with his cloud provider, I would strongly recommend having a secret scanner. So TruffleHogsfree, use it. It's open source. Or GitGuardian, both are partners of ours.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
The other big thing is outside of secrets, I would have a vulnerability scanner, really important on all your major dependency streams. So like Docker images, Ruby gems, if you're using Node, NPM, but basically any dependency you have, you should have vulnerability scanning on it. Super fricking important.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
because it's so easy these days for like one dependency that nobody's cared about for 10 years to just get bought for like a couple hundred bucks and then someone pushes an update for it um and all the other dependencies haven't frozen on that older version and so now all these dependencies get that update which means you get the update and that update has malicious code in it and that's like a very common like paradigm for attack these days
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Uh, so a freeze your, um, uh, your, uh, your dependencies, meaning like I'm fixed at this version and no amount of them pushing a new update will, will change that I'm on this version. And this version has been like, uh, guaranteed good. No, uh, no vulnerabilities in it.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And that at least can help prevent any issues where like a vulnerable code gets pushed up and you guys immediately get attacked from it. What you really want is you want all your dependencies frozen. And then when a new exploit happens, you go, oh, that was kind of cool. We are using that package, but our version doesn't have that problem. So we're good.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And then you just wait for the fix to come out and then you can do another refresher, another update.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah. So, uh, At the end of the day, Doppler makes your life better in a couple of ways. One, it's going to add two hours of productivity per developer per week. So in a way, it kind of pays for itself. On top of that, it's going to make you far more secure from an organization standpoint, managing your secrets. You can sign up today. by creating accounts for free.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And then if you're a much, much larger org, you can talk to our solutions engineering team who can help figure out how this integrates into your complex infrastructure. We have a whole host of great companies that are using us. So you're not alone from Accenture, Crumble Cookies, Puma, Children's Cancer Institute, StockX. travel trip advisors. So there's a number of great companies.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
You won't be alone in this. And we manage about 30 billion secrets a month right now. So like at quite large scale. And I think the key thing here, if you decide not to use Doppler is at least manage your secrets. Like at the end day, I think we're all tired of data breaches happening because they keep impacting us and they impact your customers.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so if you're not going to use Doppler, that's totally fine by me. You can even use a competitor, just use a secrets manager, protect those secrets.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Sounds like you guys are pretty well set up. I mean, that's the dream is to make Secret management is super easy and simple, but also very secure at the same time.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Hey, all. It's great to be here. Yeah, Doppler is a secrets manager designed to be your single source of truth for secrets across all your projects, environments, team members, and infrastructure. So you can kind of think about it like it's get up for secrets. Right.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
I think I'm a huge Star Wars fan, so I'll just shout out Star Wars Clone Wars. I feel like not a lot of people know about it, but it's between Episode 2 and 3, and it's so freaking good. I'm a guy that usually doesn't like animated stuff, and it's just blown me away. So if you're a big Star Wars fan, there's a lot of seasons, and it's insanely good, and it's basically like Dave Filoni at his best.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Outside of that, really big fan of the Perplexity app. It's basically replaced any Google searches I do now. For anyone who doesn't know what Perplexity is, it's Google search plus ChatGPT. So ask it a question, and it just gives you the knowledge back. It doesn't just give you a bunch of links. And just like a Google search, it's updated in real time.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So if something comes out yesterday or earlier today, it'll be informed on that. And it is just an incredible job at answering really in-depth questions. Like you can ask it very, very complicated questions that have a lot of nuance that have a lot of requirements on real time data.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Like, I don't know, what's the compare the market cap of Nvidia to its revenue and then compare that to Microsoft's and figure out which one is a better stock to invest in. Like you can ask that question. I'll give you a pretty concrete answer with like charts and graphs included.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So very impressed with like I think that's like a pretty big step up and the ability to like get knowledge very, very quickly. and then the other one is John Coogan I think he does a great it's a YouTube channel that he that's out there and he just does an incredible job of like talking about, I think, tricky topics.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Um, so, uh, like what's happening with Taiwan and chips or, and your old, uh, prime defense, like he just does a good job of telling the story of some event or some company really, really well. And I feel like it's extremely fact-based and not like bias-based. Um, so I, I generally go to him when I'm looking for like really high impact news, uh, breakdowns. Um,
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah, so I think Doppler kind of comes from a first principle standpoint about what does a good secrets manager look like? And for us, we have a couple of things that we deeply care about. First is that all your secrets are centralized in one place across your projects and environments.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And then if you're also a Star Wars fan, I'll just end with this again because, again, huge Star Wars nerd here. There's a board game called Star Wars Rebellion. It's played with two players. One guy gets to play as the Empire, the other as the Rebellion, and you guys battle it off, and it's quite fun.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
It's a very asymmetric game, but I think that's what makes it fun because Star Wars is very asymmetric. The Empire had a lot more resources than the Rebels. But if you like Star Wars, both fun wars and that board game will do you a lot of good.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah. I'd say me and my brother, we play, it took us eight hours, the first game to get it. The second game, we actually did it across two days just because like we were, we were kind of like very slow with every decision and be really methodical. And now we'll play like in an hour and a half. So like once you get comfortable ramps up pretty quickly.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Brian at Doppler.com for email. Balalunga Brian for LinkedIn and Twitter. Cool.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And once they're all in one place, we can set up pretty robust access controls, either by users or by groups that are policy driven. And then from there, you get a complete audit story.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So, you know, every time someone has read a secret, written a secret, changed some settings on every device with their IP address, you get a whole picture about everything that's going around or going on around your secrets. And then the other key part is that it's deeply connected to their workflows and their infrastructure.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So on the workflow side and like local development, we want you to be able to edit your secrets side by side with your code. So you have a VS code extension. And so those secrets actually kind of look like a YAML file, but they're actually not. It's like something called a virtual file system. So it behaves like a file, but it's actually not as connected with Doppler in the back end.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So it's super secure. And then on the orchestration side for production is we will write into AWS Secrets Manager and Kubernetes or wherever it may be so that your infrastructure immediately gets those secrets when they change and restart your deployment so you don't have any downtime.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
I am too, for that matter. You got to make it like we have this theme at Doppler and it's quite simple. It's make vegetables taste like candy. Vegetables being the security, candy being the developer productivity. If you do not make it, if the tool does not make the developer's lives better, like you can't just say, hey, use this tool because it's more secure.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Like that doesn't fly because developers care about moving fast and shipping product. And so you have to aid them in that. Like in our case, we try to get them two hours a day, or sorry, two hours a week back in productivity. And so if you do that, if you give them good candy, then you'll get the vegetables too. Yeah, that makes sense.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So I like to kind of bucket the risk in, in, or sorry. Yeah. Bucket the risk into two buckets. The first is like the risk to the company. And the second is the risk to people. And I think the company side is like pretty well understood, right? You're going to have a pretty, if you have a data breach, you're going to have a pretty big brand hits and trust hits.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
You're going to probably have some unexpected spend because now you're investing in legal, PR, insurance. You may get sued as well because if you're not managing your secrets and the secrets are things that get leaked that leads to a data breach, that's just straight up negligence and the customers are in their rights to sue.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
But I think the thing that's most of the time not talked about is like the cost to real people. At the end day, all these companies that we use, we're trusting them to store our data and keep our private data private. Imagine if like every YouTube video that you ever saw got out or every ride you ever took on an Uber, every transaction you ever made on a credit card all got out.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And I actually have a real personal story of how this has impacted me. Sadly, I've been part of a couple of data breaches now from some of the services I've used. And that came back and really bit me in an interesting way. So I just moved to Austin, Texas a couple of months ago from California.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And I convinced my mom to fly out with me and see Austin for herself when we were at this barbecue joint. And while we're eating food, I get a call from, uh, the Texas customs and borders. And they say, Hey, we found a package in your name that has, uh, illegal money and drugs in it. And we're, uh, federally investigating you now.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And this is like the phone call for me where I was like, Oh shit, my life has just ended. Like from the highest, really, really quickly. Um, and I was really scared and, but like also kind of like, this is odd, like never bought anything from Mexico before. So like, okay. Um, And the back of my head, I was like, this is a scam probably.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
But then they started rattling off all this information about me that they knew. All the places I had lived in the past, locations I had been to, like, just so, like, a wealth of information about me that I was like, no one could possibly know this but the government. And so, like, I trusted them. I was like, okay, I'm actually being investigated here.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And obviously we get lawyers on the call as well. Like, this becomes a really scary ordeal for all of us. And it wasn't until about like an hour in that our lawyers picked up that it wasn't that it was a scam. And but during that hour, we also gave them a whole lot more information about me. Right. Because we thought we were being investigated and they were asking questions.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so like this is and like, again, I'm a CEO of a cybersecurity company. I'm trained on this. Our employees are trained on this. Our legal team is trained on this. And they still got us for an hour. Right. I can imagine everyday Joe who's not trained could really get attacked here.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And like, I was lucky that all they did was get a little bit more information about us, but they were able to get that information because of other information that got breached. Right. That bought them the credibility to get new information.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
But imagine my mom, my sister, or anyone else like that who's not gone through any of that training, does not have legal counsel immediately readily available to them that can jump on the call for this. Oh my gosh, I cannot imagine all the information that would be given up and all of a sudden their bank accounts are drained or their credit score is completely ruined or whatever it may be.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
um it can get really terrifying really fast and that's why i come back to like whenever i see like a data breach and it's like x millions of people's data was just got out it's like wow like scammers are now like parading in the streets going like oh my gosh i just got a ton of data that's going to get me so much more data that can eventually get me to like the jackpot of all this money or or whatever they're trying to go after um and so there's real people's lives including mine that have been impacted by this and that is something that's like often not talked about at
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And so when you're a developer and you're like, oh, should I put this in a new file or not? Or should I secure my secret? It's just one token. It's just a string. It's like, well, that string may impact a million people's lives someday. So to treat it right.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
No, I think you're totally right. I mean, there's two common things that we see all the time. One is like some source code goes public or some email gets leaked and that has a secret in it to like an AWS account. And you'll see one or two things happen immediately.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
The first is their bill just like skyrockets in seconds because now they're using AWS for some other bot attack or for mining crypto or whatever it may be. That happens very, very commonly. And it's surprisingly how fast those secrets get found. I think there was an average stat that it's within three to five seconds of a repo going public.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Yeah. Outside of API keys, like a Stripe token, database URLs, encryption keys are typically the ones that we see the most. I also just have a general thought process around this of anything that's configured by the environment should be treated as a secret, so it should be treated like the most sensitive thing. So we recommend to our audience that
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
If it has a secret in it, even in the history, it will get found and start getting exploited. So it's very quick. It's insanely quick. And then... The other side of it is now that they have access to your AWS, they can start scanning all your resources. Okay, what execution environments are set up? What databases are set up? Can I get access to those databases? How privileged is this credential?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
And you start having this ripple effect where it's like, oh my gosh. Now, especially if it has like strong super admin permissions, it can start creating other credentials. And now you have this like cat and mouse game of like, how do I get out of the system? Could it lock me out of the system? Are there back doors it's placing in? So even if we remove it, it's still actually there, right?
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
it gets scary really quickly. Once they get into the system, it's very, very hard to get them out.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
Based on the data that we have from Doppler, we can't see customer secrets at all, but we can get an auto report of the names of the secrets they use if they give us that permission. And from the data we have, it's like 90% of the sensitive secrets that we put in Doppler and customers have an ability to say, this is sensitive or this is not, is actually like access tokens.
Ruby Rogues
Secrets Management Best Practices and Tools - RUBY 655
So it's not encryption tokens. Um, so it's like, if you get that Stripe key, you have it all. Like you have every customer's credit card and bank account in there. There's no extra layer of protection.