Challenges
Defend a real DevOps decision against a senior engineer, then get scored against a verified rubric. No memorizing, just reasoning.
Pick a real call
Each challenge is a true-to-life DevOps decision with a clear question to answer.
Defend it
A senior-engineer AI pushes back on your reasoning. Hold your ground or adjust.
Get scored
You are graded against a verified rubric, so you learn what a complete answer covers.
$39 one-time unlocks every challenge, the mock interviews, and the private Discord. Sign in to get access.
Beginner
161.6GB Docker Image, 8-Minute Builds. Add Runners?
Your image is huge and every code change rebuilds from scratch. A teammate wants to just add more CI runners. Defend how you actually fix it.
~8 minUnlock to play502 Bad Gateway Right After a Deploy
The site returns 502 from nginx after a deploy. A teammate wants to tweak the nginx config. Defend where you actually look first.
~8 minUnlock to playA Contractor Needs DB Access for a Week. Open the Port?
An external contractor needs a week of database access. A teammate wants to open the DB port to the internet with a strong password. Defend the safe alternatives.
~9 minUnlock to playAccess Keys Are Hardcoded on the EC2 Box. Just Rotate Them?
An app on EC2 authenticates to AWS with long-lived IAM user keys in a config file. One leaked. Defend the real fix.
~10 minUnlock to playCommit terraform.tfstate to the Repo So Everyone Shares It?
A teammate wants to commit terraform.tfstate to git so the whole team shares one state file. Defend the correct way to share Terraform state.
~10 minUnlock to playDeploy Fails with Permission Denied. chmod 777 Everything?
A deploy fails with permission errors and a teammate wants to chmod -R 777 the app directory so it never happens again. Defend fixing the actual ownership mismatch instead.
~8 minUnlock to playDeploys Are SSH + git pull at 5pm Friday. Why Change It?
The team ships by SSHing into prod and running git pull, often at 5pm Friday. A teammate says it has always worked. Defend what a real deploy needs without cargo-culting.
~9 minUnlock to playDisk Is at 100% and the App Is Down. rm -rf?
A server is out of disk and the app is down. A teammate wants to rm -rf the logs to free space fast. Defend what you actually do first.
~8 minUnlock to playHistory Is Messy, So Force-Push a Clean main?
A bad merge left main tangled with merge and revert commits. A teammate wants to rebase locally and force-push a clean history. Defend what you actually do.
~8 minUnlock to playNightly Backups Run. Are You Actually Covered?
Automated backups have run for months. A teammate says you are fully covered for disasters. Defend whether you agree.
~8 minUnlock to playTests Pass Locally, Fail in CI. Just Retry?
Your suite is green on your laptop but red in CI. A teammate wants to add a retry and re-run until green. Defend what you actually do.
~8 minUnlock to playThe Build Is Slow, So Let's Just Add --no-cache to CI?
CI Docker builds crept to 11 minutes and once shipped a stale dependency. A teammate wants --no-cache on every build. Defend what you actually do first.
~9 minUnlock to playThe Cert Expires Tonight. Just Bump It to a 10-Year Cert?
Your nginx TLS cert expires tonight and an expired cert took the site down last quarter. A teammate wants a 10-year self-signed cert. Defend the real fix.
~10 minUnlock to playThe t3 Instance Got Slow. Just Size It Up?
An app on a t3 instance crawls under steady load and a teammate wants a bigger instance. Defend what is actually throttling it.
~9 minUnlock to playYou Pushed AWS Keys to a Public Repo
Live cloud keys just landed in a public GitHub repo. A teammate says delete the file and force-push. Defend what you do first.
~8 minUnlock to playYour S3 Website Returns 403. Make the Bucket Public?
Objects in S3 return 403 and a teammate wants to turn off Block Public Access to fix it fast. Defend the right way to grant access.
~9 minUnlock to playIntermediate
203 AM Outage: Roll Back or Fix Forward?
A deploy 20 minutes ago correlates with a spike in 500s. You are on call. Defend whether to roll back or fix forward.
~12 minUnlock to playA Traffic Spike Caused an Outage. Autoscaling Did Not Save You.
A spike took the service down even though autoscaling was on. A teammate wants to crank the max replicas. Defend the real fix.
~13 minUnlock to playAdd a NOT NULL Column to a 50M-Row Prod Table
A migration adds a NOT NULL column to a huge live table. A teammate wants to run the ALTER directly. Defend the safe path.
~14 minUnlock to playAfter an Outage, Mandate Manual Approval on Every Deploy?
After a bad deploy caused an outage, leadership wants a mandatory human sign-off on every production release. Defend a better way to prevent the next one.
~13 minUnlock to playChasing a Bug: Flip Production to DEBUG and Log Everything?
To catch an intermittent bug, a teammate wants to set the whole prod app to DEBUG and log full request and response bodies. Defend a safer approach.
~12 minUnlock to playDeploy From CI: Paste AWS Keys Into GitHub Secrets?
Your GitHub Actions pipeline needs to deploy to AWS. A teammate wants to paste a long-lived IAM access key into repo secrets. Defend the better option.
~12 minUnlock to playEvery Deploy Hammers the Database for 10 Minutes. Bigger DB?
Each deploy flushes the cache and the database melts for 10 minutes while it refills. A teammate wants to double the database size. Defend fixing the stampede instead of paying for peak.
~12 minUnlock to playFinance Wants Everything on Spot Instances. Including the Database.
Finance wants the whole fleet, primary database included, moved to spot instances for 70 percent savings. Defend what actually belongs on spot and what a reclaimed primary really costs.
~13 minUnlock to playLambda Is Exhausting Your RDS Connections. Raise the Limit?
Under load, Lambda functions fail with too many connections to RDS. A teammate wants to bump max_connections. Defend the real fix.
~12 minUnlock to playOn-Call Is Drowning in Alerts. Silence Them?
A CPU alert pages constantly with no real impact. A teammate wants to silence it. Defend how you fix alerting properly.
~12 minUnlock to playPod Stuck in CrashLoopBackOff After a Deploy
A new deploy leaves a pod in CrashLoopBackOff. A teammate wants to bump CPU and memory. Defend how you actually diagnose it.
~12 minUnlock to playPods Keep OOMKilling: Scale Up or Fix the Leak?
A service OOMKills every few hours. Adding replicas or raising memory limits buys time. Defend whether to scale around it or fix the root cause.
~12 minUnlock to playSomeone Changed Infra by Hand. Terraform Plan Is a Mess.
A manual console change put Terraform's state out of sync, and the next plan wants to undo it. Defend how you reconcile it.
~12 minUnlock to playSQS Is Delivering the Same Message Twice. Is the Queue Broken?
Consumers occasionally process the same SQS message twice, causing double charges. A teammate says switch everything to FIFO. Defend the real fix.
~13 minUnlock to playTerraform Doesn't Know About the Manually-Made DB. Just Let It Recreate It?
A prod RDS instance was made by hand and Terraform now wants to destroy and recreate it. A teammate says just apply. Defend what you actually do first.
~12 minUnlock to playThe Container Needs Docker, So Mount the Socket and Run --privileged?
A CI job needs to build images and a teammate wants to bind-mount the Docker socket and run --privileged to fix permission errors. Defend the safe approach.
~12 minUnlock to playThe Fix Is One Line in /etc. Just SSH In and Edit It?
Prod is degraded by one wrong sysctl value in /etc on an Ansible-managed fleet. A teammate wants to hand-edit each box now and codify it later. Defend the call.
~11 minUnlock to playThe NAT Gateway Bill Exploded. Add More Gateways?
NAT Gateway data processing is now your biggest line item. A teammate thinks it is just bandwidth. Defend where the cost really comes from.
~12 minUnlock to playThe Pipeline Is Slow. Skip Tests to Ship?
A 40-minute pipeline is blocking releases. A teammate wants to skip the test stage to ship faster. Defend your call.
~12 minUnlock to playThe Queue Backlog Doubles Every Day. 10x the Consumers Tonight?
A consumer queue backlog is doubling daily and a teammate wants to 10x the consumer fleet tonight. Defend diagnosing the real bottleneck first, and why blind consumer scaling can make the outage worse.
~12 minUnlock to playAdvanced
17A 30-Second Blip Became a 2-Hour Outage. More Retries?
A brief downstream blip cascaded into a full-platform meltdown because every layer retried. A teammate wants to raise retry counts. Defend the real fix.
~14 minUnlock to playA Container in Prod Is Mining Crypto
A pod is pegging CPU running a crypto miner. A teammate wants to kubectl delete it and move on. Defend the real response.
~15 minUnlock to playAn EKS Pod Needs AWS Access. Attach It to the Node Role?
A pod needs to read S3, and a teammate wants to add the permission to the EKS node role to unblock it. Defend the least-privilege approach.
~15 minUnlock to playCross-Account S3 Access Still Fails. Just Open It Up?
A partner account still gets AccessDenied on your encrypted S3 objects after you added a bucket policy. A teammate wants to make the bucket and key public. Defend the right fix.
~15 minUnlock to playDynamoDB Is Throttling but Capacity Looks Idle. Crank It Up?
DynamoDB throttles some requests while table capacity looks underused. A teammate wants to raise capacity or flip to on-demand. Defend the real cause.
~15 minUnlock to playFailover Lost 5 Minutes of Writes. Promise Zero Data Loss?
An async-replica failover dropped five minutes of committed writes. Leadership demands zero data loss with zero performance impact. Defend the honest trade-off.
~14 minUnlock to playOne Giant Terraform State Runs Everything
A single Terraform state holds all your infra; every plan is slow and scary. A teammate wants to just keep applying carefully. Defend how you reduce the blast radius.
~15 minUnlock to playp99 Latency Spiked Across a Microservice Chain
p99 latency jumped across a request that touches six services. A teammate wants to scale up the slowest-looking one. Defend how you find the real culprit.
~15 minUnlock to playSecrets Rotation Took Down Prod at 3 AM. Pin Credentials?
Automated rotation revoked a database password while pools and caches still held it, locking prod out at 3am. A teammate wants to pin credentials for a year. Defend fixing rotation instead.
~13 minUnlock to playService A Intermittently Cannot Reach Service B
One in twenty calls between two services fails. A teammate wants to add retries and move on. Defend how you actually find the cause.
~15 minUnlock to playTerraform Wants to Replace the Database. Apply Now?
A routine terraform plan unexpectedly wants to destroy and recreate the production database. Defend whether to apply, and how.
~14 minUnlock to playThe Box Keeps OOM-Killing. Just Add a Huge Swapfile?
A latency-sensitive service keeps getting OOM-killed and a teammate wants to add 16GB of swap and crank swappiness. Defend what you actually do instead.
~13 minUnlock to playThe Nightly Report Is Slow. Just Run It in One Big Transaction?
A nightly report holds one Postgres transaction open for hours and writes are bloating. A teammate wants a bigger box and higher timeouts. Defend the real fix.
~13 minUnlock to playUnder Load, Healthy Pods Keep Restarting. Add Replicas?
During traffic peaks, Kubernetes keeps killing pods that are not actually broken. A teammate wants to scale up replicas. Defend the real fix.
~14 minUnlock to playWe're Missing a Dimension, So Add user_id as a Label?
Debugging lacked per-user detail, so a teammate wants user_id and request_id as Prometheus labels. Prometheus is already under memory pressure. Defend the call.
~14 minUnlock to playYour CI Has Keys to Prod and Pulls Unpinned Deps
CI pulls unpinned dependencies, runs them, and holds prod credentials. A teammate says just add a vulnerability scanner. Defend a real hardening plan.
~15 minUnlock to playYour Primary Region Is Down. Fail Over Now?
The primary region is degraded. A teammate wants to flip DNS to the standby immediately. Defend whether and how you fail over.
~15 minUnlock to play