I Built an AWS Infrastructure Lab — Then I Broke It on Purpose
I Built an AWS Infrastructure Lab — Then I Broke It on Purpose
When I started working seriously with AWS, I didn't want to spend my time memorizing services. I wanted to understand what was actually happening. What happens when an EC2 instance sends a packet to the Internet? Why is one subnet considered public while another one is private? Why can two machines communicate inside the same VPC even when one of them has no public IP? What actually happens when a route disappears? And when something breaks, how do I find the reason instead of just restarting things until they work? So I decided to build a small AWS environment with Terraform and use it as a laboratory. Not a production system. Not a giant architecture with every AWS service possible. Just enough infrastructure to make me think like an infrastructure engineer. And then I started breaking it.
Starting with the network
I built a VPC with the CIDR: 10.0.0.0/16 Inside it, I created two subnets: Public subnet 10.0.1.0/24 Private subnet 10.0.2.0/24 The public subnet contains an EC2 instance with a public IP. The private subnet contains another EC2 instance without a public IP. At first glance, this sounds straightforward. But I wanted to understand why the two subnets behave differently. The public subnet has a default route: 0.0.0.0/0 → Internet Gateway The VPC also has its automatic local route: 10.0.0.0/16 → local That one distinction explains a lot. The public instance can reach the Internet because there is a path from the subnet to the Internet Gateway. The private instance can communicate with other resources inside the VPC because of the local route, but it doesn't have a path to the public Internet. That distinction became much clearer once I stopped looking at AWS as a collection of services and started looking at it as a network.
I wanted to see what would happen if I broke the network
This was probably one of my favorite experiments. I deleted the default route from the public route table. Basically: 0.0.0.0/0 → Internet Gateway was gone. Then I tried to SSH into the EC2 instance. It failed. That sounds obvious after the fact, but this was exactly the kind of experiment I wanted. The EC2 instance itself was still running. The operating system was still running. The security group hadn't suddenly disappeared. But the network path was broken. So I restored the route: 0.0.0.0/0 → Internet Gateway and SSH worked again. That gave me a much better mental model of troubleshooting. When a machine isn't reachable, don't immediately assume: "The server is down." There are many layers between you and that server.
Then I tried the private instance
The private EC2 instance had no public IP. I connected to it through the public EC2 instance. The path looked like this: My laptop | v Public EC2 | v Private EC2 The interesting part was that I didn't need to give the private instance a public IP. Both instances were inside the same VPC. The local VPC route handled communication between them. I also used SSH agent forwarding so that I didn't have to copy my private SSH key onto the public instance. That was a small detail, but it reinforced something important: private networking doesn't mean isolated from everything. It means the connectivity is controlled differently.
The private EC2 couldn't reach the Internet
Then I tested: curl -I https://www.google.com from the private instance. It timed out. At first, that might look strange because the operating system had a default route. But the operating system having a route doesn't mean AWS has a complete path to the Internet. The machine could route traffic toward its subnet gateway, but there was no NAT Gateway providing Internet access for the private subnet. So the packet had nowhere useful to go beyond the VPC. That was one of those moments where the architecture diagram finally matched what I was seeing from the terminal.
Adding storage
I also wanted to understand EBS beyond simply knowing that it is "AWS block storage." I created a small: 2 GiB GP3 EBS volume and attached it to the EC2 instance. Inside Linux, I formatted it with XFS: sudo mkfs -t xfs /dev/nvme1n1 Then mounted it: sudo mkdir /data sudo mount /dev/nvme1n1 /data I created a simple test file: echo "Hello from Ayoub's EBS volume" | sudo tee /data/test.txt Then came the interesting part. I detached the volume and attached it again. After mounting it again, the file was still there. That sounds simple, but it demonstrates an important concept: the EC2 instance and the EBS volume have different lifecycles. The instance provides compute. The EBS volume provides persistent block storage. Once you see that distinction, a lot of AWS architecture starts making more sense.
Giving the EC2 instance an IAM role
I also wanted to avoid the bad habit of putting AWS access keys directly on a server. So I created an IAM role and attached it to the EC2 instance through an instance profile. For this lab, I gave it a small S3 permission: s3:ListAllMyBuckets The important thing wasn't the S3 operation itself. It was understanding the relationship: EC2 ↓ Instance Profile ↓ IAM Role ↓ AWS Permissions The application doesn't need a long-lived access key sitting somewhere on the filesystem. That is a much better mental model for workloads running inside AWS.
Monitoring the machine
Once the infrastructure was working, I didn't want to stop at: "Everything is green." Real infrastructure needs to be observed. I started exploring CloudWatch EC2 metrics. Some of the metrics available included:
- CPU utilization
- Network traffic
- Network packets
- EBS read/write operations
- EBS read/write bytes
- Status checks
- CPU credits
I queried CPU utilization from the AWS CLI and saw the instance sitting at a relatively low baseline. Then I decided to create a problem.
I intentionally created a CPU incident
I ran: yes > /dev/null & The process continuously generated CPU activity. Then I watched CloudWatch. The CPU utilization increased. That was the easy part. The interesting part was figuring out what I would do if this happened on a real machine and I didn't already know the cause. I started investigating from the operating system. top Then: ps aux --sort=-%cpu I also checked: uptime free -h and system logs with: sudo journalctl -n 50 --no-pager Eventually, the CPU-consuming process was obvious. I terminated it: pkill yes Then I checked the metrics again. The CPU usage started returning toward normal.
Turning the incident into an alert
I didn't want to rely on manually staring at CloudWatch. So I created a CloudWatch alarm: CPUUtilization > 20% with a five-minute evaluation period. The basic flow became: EC2with me a blog in ↓ CloudWatch ↓ CPU metric ↓ Alarm ↓ SNS ↓ Notification This changed the exercise from simply: "I can see CPU utilization." to: "The infrastructure can tell me when something abnormal happens." That's a much more useful way to think about observability.
The bigger lesson: monitoring isn't troubleshooting
One thing I took away from this experiment is that metrics don't necessarily tell you the whole story. CloudWatch can tell me: CPU utilization is high. But it doesn't automatically tell me: This specific Linux process is responsible. For that, I need to go onto the machine and investigate. That's where tools like: top ps journalctl free df ip become useful. So I started thinking about monitoring in layers. CloudWatch tells me something is happening. Linux tools help me understand what is happening. And eventually, with a larger distributed system, traces can help explain where a request went across multiple services.
Why I used Terraform
I could have created all of this manually through the AWS Console. But that wasn't really the point. I wanted the infrastructure to exist as code. So the environment is managed with Terraform. The workflow became: terraform init ↓ terraform validate ↓ terraform plan ↓ terraform apply And when the lab is finished: terraform destroy Terraform also forced me to think about infrastructure differently. Instead of: "I clicked these buttons and created this." I have: "This is the desired state of my infrastructure." That is a much more powerful way to work.
I also learned from things that went wrong
Not every lesson came from successfully deploying something. For example, I initially learned the hard way that automatically selecting the newest AMI can make infrastructure less predictable. So I pinned the AMI used by the lab. I also observed that an EC2 public IP can change after stopping and starting the instance. And I learned that Terraform can sometimes want to replace resources when an important underlying attribute changes. These details aren't particularly exciting in an architecture diagram. But they're exactly the kind of details that become important when you're responsible for infrastructure.
What this lab actually gave me
The final infrastructure isn't huge. It's essentially: AWS │ └── VPC │ ├── Public Subnet │ └── EC2 │ └── EBS │ └── Private Subnet └── EC2 EC2 │ └── CloudWatch │ └── Alarm │ └── SNS But I learned much more than the number of resources suggests. I worked through: VPC networking Public vs private subnets Route tables Internet Gateway Security groups EC2 SSH SSH agent forwarding IAM roles EBS Linux storage CloudWatch SNS Terraform CPU investigation Failure injection Incident response More importantly, I practiced a way of thinking.
Build it. Break it. Understand it.
I think this is the part that matters most. It's easy to follow a tutorial where every command works. You type something. It works. You move to the next section. At the end, everything is green. But infrastructure engineering doesn't work like that. Things fail. Routes disappear. Processes consume CPU. Instances become unreachable. Storage gets detached. Deployments behave differently from what you expected. And sometimes the most useful thing you can do is deliberately break something in a controlled environment and find out why. That's what I wanted from this project. Not just: "I know AWS." But: "I can reason about the system."
What's next?
This lab is only the beginning. The next step is to make the infrastructure more production-oriented and go deeper into the engineering behind it. I'm moving toward:
- Production Terraform
- AWS architecture
- Linux and systems engineering
- Kubernetes
- Kubernetes networking
- Observability
- Prometheus and Grafana
- Infrastructure automation
- Go for infrastructure
- CI/CD and platform engineering
Eventually, I want to turn these individual skills into larger systems that solve real problems. For now, though, I'm happy with something much simpler: I built a small infrastructure environment. I understood how the pieces connected. I broke it. I investigated the failures. I fixed them. And I documented what I learned. That's exactly the kind of foundation I want to build on.