AWS Journey

Last Edited On: November 6, 2023 @ 5:49 PM

Well since scalability and reliability are becoming more and more important the used of the cloud are being more and more popular. There are a lot and I mean a lot of things that you can do with in AWS that it can look daunting at times, well that is how I felt not too long ago before I just dived into it. I am going to break it down into steps on what I learnt.

VPC & EC2

I had some previous experience with the cloud but it was mostly just in Azure and not AWS so it was nice to see that it was familiar but everything in AWS have their own names for things like EC2s they are just VMs(virtual machines).

Starting at the base level was the VPC it is basically the network of which you are going to be constructing your whole network. All I am going to say is that if someone does not have any knowledge about networks how they are made, managed, created and so on they would struggle.

Once I created the VPC I was still off put by the interface that AWS was it was like I just opened up an IDE for the first time.

Moving to the EC2s I have created 2 instances of the lowest tier possible since it was the easiest way to get a network setup. I wanted to create a managed network with a DC(domain controller), AD(active directory) and DHPC Server. Being the absolute newbie that I was I had knowledge on how to run them and maintain them but never have I ever set them up from scratch. I would say it was a pleasant experience, building nothing to something that accepted users.

After all of the setup I was hooked on AWS and how easy it was to setup technically anything it felt like my inner child just got a hold of a new Lego set but it had no limit and the only thing that was limiting myself was my knowledge on how thing are built, of course there is the aspect of how expensive things can get.

Once I had some ec2 instances running and they were all connected into a domain and talking to each other it was next to make something that would have an actual functionality.

Code Pipeline

The concept of using a CDCI(continues development continues integration) pipeline was a REALLY REALLY nice thing I have experienced previously when using Vercel. When you push your code to the main branch it instantly builds the code and publishes it. I though that I can do that as well so why not just do it.

It is a no brainer that in modern development you don't want to worry about small things like were to deploy and ssh’ing into a server to pull the git repo that has the code and all of that mess. So learning how to create a CICD pipeline in AWS was the way to go.

⚠ OO boy was this a lesson

I decided that I will use .ASPNET for the framework that I will be using for my use case. I was familiar with ASPNETCORE so I just made the project nicely it comes with a weather API that I could use for testing, one important thing is that the ec2 instance that this was going to be on was Linux based. The next steps were to create a service that the application would run on and how it would be launched, struggled a bit since the application was set to localhost and would not run had to change it to the 127.0.0.1 ip to actually make it work and some other tweaks had to be made for the code to work from a ec2 instance.

So what we have right now is the API that is connected to the DB which is in AWS but the problem now is database as it was not a replication but just a snapshot of the data so the API was responding with historical data. To change this a stream between the two databased were need to be created as the two databases would stream data between each other but the cost of that was very high so it did not go through.

S3 and Policies

Learning how to use @aws-sdk/client-s3 which is a npm package that is used for connecting AWS S3 buckets to your JS/TS environment. As of writing this post there are multiple versions that are going around of the sdk since AWS is moving forward with a newer version but the older sdk is still functional. When trying to upload documents from a UI interface to S3 there are soo many potential hurdles that can into play.

I was using nextJS 14 with the app router for this implementation.

  1. Posting the request to the backend and parsing the formdata into the sdk

I had soo many issues here when trying to get the data from a FormData type to input the files into the sdk, there were some errors that said that it only takes in Blobs and not Files types and so on. The solution I got to was to convert the file to a base64 and just post it with the sdk.

  1. Policy issues this was direct in aws with roles and policies

The scenario I had is that I needed to list all of the files that was in S3. The issue was the response, you do not have the permissions. The issue was that the permissions of the policy that was attached to the user that had the access key. It stated that I had the permission as the first one and changed it to the second one.

bucketName/ —> bucketName/*

that was the issue small but very meaningful, it meant I could list all of the sub directories that are in the bucket but not the directories in the bucket itself. After that fix in the policy I was able to get all of the directories in the bucket.

Yes, I know how S3 stores their objects they technically do not get stored in directories along with the fact that the ‘files’ are not files but objects that just contain the structure of the file.