Playing with Fire

September 4, 2026

By Rare Hydrogen

Or, my theory of Web Application Security on ephemeral Virtual Machines

Introduction

NOTE: I’m writing this for myself, because I think about this stuff alot and I’d like to compress some knowledge the old-fashioned way; with words.

I like building services and apps on virtual machines. I like the challenge of maximizing utility for a fixed monthly cost. In general, I prefer fixed costs to variable costs because I value predictability. There is also a great degree of freedom allowed by virtual machines, since they are basically the building blocks of the cloud, and by building with them, I think I’ve gained a stronger understanding of the entire stack. I also like being able to jump right into any box with a single terminal command, thanks to SSH. With other infrastructure, I have to find Docker ids or use proprietary consoles and UI to debug things, and I don’t like jumping through extra hoops. I prefer direct, uniform and simple.

That’s why I like building with Linux VMs as opposed to something managed like App Engine or Vercel or whatever PAAS is the flavor of the month.

Stage 0: Protect your box

Before you put anything sensitive on your shiny new VM, you should take some time to secure it. Unlike PAAS solutions, working with VMs means you have to care about what you’re doing. That “care” often saves you money and helps teach you things!

Exposing a machine to the world wide web opens it up to attack! Baddies from all over the world are sniffing out connected IPs for targets and running automated attacks against them. We can reduce these attacks by implementing some simple security measures:

Use a reverse proxy:
- Block unwanted traffic (nobody in Russia needs to see my app).
- Expose multiple services behind a single endpoint.
- Handle SSL in one convenient place. (BTW SSL is non-negotiable!)
Use a STRONG reverse proxy:
- Cloudflare CDN is so big and tough it can shrug-off DDOS attacks.
Disable Password logins for your users
- Instead of logging in with a password like you would on your laptop in the early 2000’s, you’ll login by connecting with SSH.
- SSH uses the private-public key pair to make the connection.
- This is safer because a password can be cracked a lot faster than a .PEM file.
Install and enable Fail2Ban
- This software keeps track of how many times an IP tried to login.
- If they try more than x number of times from an IP, that IP is banned.
- F2B keeps track of banned IP and refuses their connections.
Depending on your app, if an external service triggers it to startup (Github Actions / CI/CD process), you may need to do the following:
- Make a “Deploy” user. (Don’t use your root admin!)
- Give the Deploy user its own SSH connection.
- Give the external service the SSH connection, so it can any script on your box.
- Limit the scripts the deploy user can run with an entry in the Sudoers file.
- Enable your application to be run by the Deploy user (if needed).
Disable PTRACE_ATTACH
- This prevents attackers from putting debuggers on your process and reading secrets out of your application

Stage 1: Store credentials on VMs in plain text

When your app needs an environment variable or API key, the easiest path is to just plop it in the server. Simple, right? You need to modify your app so it reads a file on startup, or throws an error if it isn’t there. But that logic is easy. Also, if you ever need to delete your VM and start it up on another machine (or locally to test it), having all the environment variables stored in one place is really convenient.

But wait, don’t stop here!!!

Your VM just became an even juicier target for hackers! Since all your secrets are stored in plaintext, if a bad guy breaks in, they can easily find them!

Stage 1.5: Store credentials in Memory only

Basically, the idea here is to pass your env vars to the application as part of the startup process. Then, the application only has the creds in active memory. When you do this right, it becomes a lot harder to get the secrets because they are stored in memory, not a plaintext file. Importantly, these secrets are not impossible to access by an attacker, but they are significantly harder.

This is pretty good, just make sure you’re not passing the arguments directly in the startup command! These are visible in plaintext of your command history. Instead, store them in the same environment file but hard delete the file after the application process is running. You’ll need to write more code to achieve this (bash scripts) and cache the environment variables in your app, but that’s worth it.

However, once you do this, you won’t be able to successfully restart your app without adding the secret files back temporarily. That can make things harder to manage directly from inside the VM. That may not sound like the biggest deal, and in practice, it isn’t. Whenever you want to start your app, you HAVE to use the CI/CD process you built before.

Unfortunately, your VM provider may not always be the most reliable vendor, and sometimes they might restart your VM for random reasons. This would leave your app down at 2 AM unbeknownst to you!

Stage 2: Store credentials in a Secret Manager

A secrets manager is a third-party service that asks for an access token from your app BEFORE delivering your secrets. It is also a remote, single-source-of-truth repository for your secrets. Your secrets can still be cached in memory by the application. But the only thing stored on the machine is the access token that can eventually expire, or be invalidated by the secrets manager.

A secrets manager doesn’t actually make your secrets safer from attackers. If an attacker gets root access to your box, they can still get the credentials by dumping the memory the same as they could in the previous stage.

However, a secrets manager does give you additional tools in your security toolbox:

Rotating your secrets becomes significantly faster and easier (meaning old / exposed secrets won’t hurt you anymore)
Audit logs recording every time your secrets are pulled can help you pinpoint exactly when you were breached. Knowledge is power.
If the access token is stored on the machine, restarting the app can be done as part of the VM’s startup process, alleviating host-induced shutdowns and restarts.
I’m pretty sure a range of IPs can be given to secrets manager, and if the request doesn’t come from that IP, deny the request.

With a secrets manager, your app security is more visible and its easier to maintain good practices. If I detect the error fast enough, I can disable the token and rotate the keys, and the attacker gets locked out. So stage 3 isn’t any more secure than stage 2 is, but it’s alot easier to manage and resolve problems.

Stage 3: Advanced solutions to the Secret Zero Problem

If you’re like me, you might look at stage 2 and wonder if there’s any way to secure the access token from prying eyes.

Maybe I could encrypt the token, which would be more secure, but then I have to manage storing the encryption key somewhere too. I could encrypt that key too, or store it in a different secrets manager, but then I have a second key to manage at that point, and so on and so on. As you can imagine, there’s no end to this, and this regression problem is famous enough to have a name: The Secret Zero Problem.

I believe that there are some attempts to solve the secret zero problem, but they have their own drawbacks. Hardware Security Modules can isolate secrets in a tamper-proof physical location, but that doesn’t really vibe with my VM deployment model at the moment. There are also identity-based alternatives that I haven’t investigated but that could potentially mitigate the secret zero problem, but they depend on vendor-specific implementations, and the whole point of VM-based deployment for me is to build something that is vendor-agnostic. Also, I probably don’t need the level of security they provide, because the threat model I’m facing and what I have to protect are relatively uninteresting.

That’s not really a satisfactory answer, and I’m not personally satisfied by it, but it is the best I’ve come up with so far. It’s important to know your own limitations, I guess.