AWS (Amazon Web Services)- Tutorial
What is “Cloud Computing” ?
Cloud computing is a terminology, which has the characteristics like On-Demand, Pooling ability, Faster Elasticity.
Cloud computing basically supports “AS a Service Models” like
1. IAAS (Infrastructure as a service).
2. Software as a service.
3. Platform as a service.
AWS Region, is kind of framework, in which we have all the available providers.
For EC2 instance, Storage, DB – we need a region in which we can build our services). We have 17 Regions at the time of preparing this tutorial.
For more information on Regions and Availability Zones please refer the below link:
Edge Location are the locations where end user services are provided.
Your business center is located in India and that is the reason why you have selected the Region as India, however your clients are based out of UK so we select an edge location in UK which would lend the content provided by you to the clients from the UK edge location- this will not only save the Network/bandwidth but also improves the performance of your server.
Networking & content Delivery
AWS VPC (Virtual Private Networks): VPC is a AWS Resource, through which you define and design a virtual network unlike your traditional network which is setup in the Data Centers. Since it is virtual network it is easy to scale.
AWS VPC is created with in the Region. By default each instance has a private IPv4 address assigned automatically by AWS, we also have an option to specify a public IP address, which we get it by making changes in the subnet. So we can connect to the instance from anywhere in the world via internet. With this we could also assign multiple IP’s to an instance. (* Terms and conditions apply : In order to connect to the instance from anywhere in the world we have to make sure that the internet gateway, Ingress or Egress rules have been defined properly)
General Classes in IP address (Quick Recap)
- Class A: 1 – 126
- Class B: 128 – 191
- Class C: 192 – 223
- Class D: 224 – 239
- Class E: 240 – 254
Out of which, 127 and 255 are reserved for loopback and diagnostic purpose. A part from the above two IP’s 127 & 255, AWS reserves 4 More IP address for Diagnostic purposes.
Subnets is set of IP address, which falls within the created VPC range. Unlike our traditional networking in DC we can have different sets of IP address, like public and private subnets. So using Public Subnet, EC2 resources could be connected to Internet directly.
Creating a VPC and a SUBNET
1. Always make sure you are selecting the right Region. (We have selected Ohio)
2. All the AWS services are listed in the SERVICES drop down. Select VPC under Networking and content delivery.
3. Select “Launch VPC Wizard.
NOTE: We have selected “VPC Dashboard” and then selected “Launch VPC wizard”, How ever we could also select “your VPCs” and create VPC and subnet’s individually.
4. In our example we have selected “VPC with a Single Public Subnet” you man choose as per the Need and click the “Select” button.
5. Mention the CIDR block for VPC, and subnet with in that VPC. Select the Availability Zone in which you want to create, We have selected “US-East-2a”. lastly Mention the Subnet name for ease understanding and click on “Create”. On the Next screen it will display that the VPC is successfully created and then Click on “Ok”.
6. This will take you to the VPC Dashboard. You should be able to see the created VPC by selecting “your VPC’s” and created subnets by selecting “subnets”.
3. Route Table
Routing table in AWS is same as Routing table in Linux/windows, which is a set of rules defined to route the traffic as needed. There is a default route table created, however it is always good to create custom defined route table associated to a subnet.
Each route table has CIDR and a target to the corporate network (which acts as a virtual private Gateway).
Creating a Route Table
a. From the “VPC Dashboard” select “Route Tables” and click on “Create Route Table”. As shown below, once it prompts to create a route table mention the “Name tag” and select the VPC for which you want to create the route table and click on “Yes Create”.
b. Now for the route that we created, we need to attach a subnet, select the subnet and click on Save.
4. Internet Gateways
Internet gateway is the communication between Instances within the VPC and Internet. Internet Gateway also provides a target in the VPC route tables and Internet traffic (routing).
In our case the Internet gateway has been already created. But if you do not see one, go a head and select “create internet gateway” and create one.
5. Egress Only Internet Gateways:
Egress Internet Gateways is an outbound communication over IPv6 from instances with in the VPC to the outside world.
DHCP Options Set: Dynamic Host Control Protocol (DHCP), is a configuration where “Domain Name, Domain name Server and NetBIOS-node” are defined.
6. Elastic IP’s:
Elastic IP address is an IPv4 addressing which can rapidly route the address to another instance in case of failure. Elastic IP is Static. Elastic IP as of now is not Supported to IPv6 addressing.
Endpoints:End points are virtual devices which enables communication between instances in the VPC. Also Endpoints enables to connect to the VPC privately.
NOTE: Elastic IP’s are chargeable, from the time it is generated. Irrespective of weather it is used or not.
7. Endpoints and Endpoint Services
There are two types of Endpoints:
1.Interface Endpoints:Interface Endpoints is an elastic network Interface with a private IP.
2.Gateway Endpoints:Gateway Endpoint is a target for a defined route in the route table.
Endpoint Services: Endpoint services are referred to as AWS Private Link where you can configure/create your own application in the VPC. Also using Interface VPC Endpoints we can create a connection to another VPC.
Example: Steps to create a VPC Endpoint Service for the mentioned subnet, is mentioned below.
1. Go to VPC Dashboard. Check if you are in the correct Region. Select the VPC for which you want to create the Endpoint. Click on Endpoint.
b. Create endpoint
C. Select the service for which you want to create the Endpoint for and select the VPC again. It is important that you select the Subnet for the VPC and then click on the Create Endpoint Button.
I have selected the policy as Full access, however as per your requirement you could do your custom settings.
8.NAT(Network Address Translation) :
Nat Gateway is Connectivity between private subnet and internet. Which helps in things like software updates. However which also prevents sources from Internet to get in touch with Instances in AWS.
Nat Gateway’s supports only for IPv4 and NOT available for IPv6.
Types of NAT’s:
NAT Gateway: Is Same as NAT defined Above.
NAT Instance: Used in the public subnet for the VPC. So as to get the outbound traffic from Private subnet to internet.
Creating a NAT Gateway.
a. Go to the VPC dashboard and select NAT Gateways from the menu. Now click on “Create NAT Gateway”.
b. Select the subnet and Elastic IP (if already created one) for which you want to create the NAT Gateway and click “Create a NAT Gateway.
If you do not have an Elastic IP already created then click on “Create New EIP” which will generate an Elastic IP.
As discussed before, Elastic IP’s are chargeable from the time you generate it. Weather you use them or now. Because once generated means – the IP is not getting used by any one else and is blocked by us.
Peering connection is networking connection between two VPC’s. Which makes instances in both the VPC’s to communicate with each other.Security: Security is an important feature that helps not only to enhance the security but also to monitor for the VPC.
Creating Peering Connection:
Creating Peering connection is simple, Got to VPC dashboard and select “Peering Connections” from the left panel and click on “Create Peering Connection” which will take you to a new screen where you have to name the Connection and select source and destination(vice-versa) and click on “Create Peering connection as shown below.
This is like a firewall which controls the inbound and outbound traffic at the subnet level. Network ACL’s have both Inbound and Outbound rules which has an advantage toAllow or Deny the set rule.
Eg: Some thing comes in as an initial connection and packets go out in response then NACl does not recognize. Due to which we have to specify both Ingress and Egress Rules. Each rule will have a provision to Allow or Deny
2. Security Groups:
This is like a firewall for the EC2 instances, which controls the inbound and outbound traffic at the instance level. Security Groups also have inboundand outbound rules but cannot allow/deny.
Security Groups recognizes the response of the connection, so either Ingress OR Egress any one needs to be specified.
3. Flow Logs:
Flow logs collects information of the traffic (both inbound and outbound) for the VPC.
Creating Network ACL’s(Same is the way to create security group, however as discussed we mention only one rule – inbound OR outbound)
a. From the VPC dashboard, under security select “Network ACL’s”. Give the name Tag and select the VPC for which you want to provide the security and click “Yes Create”.
b. Once the Network ACL is created, go to Network ACL’s and mention Inbound and Outbound Rules as mentioned below.
You have option to add multiple rules, as per your need. It is important to remember that the rule number should be above 100.
4. VPN Connections
VPN connections is also connectivity between VPC and remote connectivity but using third party software.
Customer Gateways is connectivity between VPC and remote Network. The Virtual Private Gateway in AWS provides two VPN endpoints for auto failover and at the remote end give the details of the customer Gateway.
Virtual Private Gateways:
Virtual Private Gateways is also connectivity between VPC and Remote network but which has multiple Remote connections.
AWS Direct Connect:
AWS Direct connect is a dedicated private physical connection from customers network to VPC. i.e AWS direct connect links the internal network to AWS Direct Connect over a Ethernet using fiber optic cable.
a. Cloud Watch: Cloud watch is a monitoring service through which we can monitor the application, resource utilization by which we can manage the health of the resources(AWS EC2 instances, storage and applications). In simple it collects and stores the logs.
How to create a Cloud Watch for an Instance if the instance reaches maximum threshold with respect to CPU.
- Go to EC2 Dashboard, select the instance.
- Go to “Monitoring” Tab and select “Create Alarm”.
3. Give the notification Name, Email ID of the team/person responsible for it and action to be taken to the instance in case of breach. In this example we have set the Maximum threshold of the CPU to 95%, and if the state of the CPU continuous to the maximum threshold for a period of 5 mins for two consecutive periods then it will not only notify you to the Given Email id but also will shutdown the Instance.
Initially the state of the alarm will be shown as Insufficient which could be ignored.
2. Aws Auto Scaling:
AWS Auto Scaling is to scale your applications automatically a to maintain steady performance of the resources.
AWS Auto scaling is to monitor the load of the system with respect to process and memory which directly depends on the load of the application and hence helps scaling the EC2 instances automatically to handle the load. Collections of EC2 instances is called Auto scaling Group.
You can specify the minimum and maximum number of EC2 instances in a auto scaling group. Also if you specify the auto scaling policy where it launches instances or terminates the instances as per the load of the server (i.e as per the demand of the application load increase or decrease)
3. Cloud Formation:
AWS Cloud Formation is a CLI tool which provides you to provision all the infrastructure resources in the cloud.
AWS CloudFormation is a service that helps to automate your AWS resources. In Cloud Formation you create a template where you mention all your resources that you want (Eg. EC2 instances) and the AWS Cloud Formation (the template designed) will help you build/provision and configure the resources.
AWS Cloud Formation is a very powerful feature by AWS which could be used as “Infrastructure as code”.
Terraforms and Packer are called Infrastructure Build tools, through which we could build AWS cloud formation templates.
AWS Cloud Trail is a service which is mainly used for Audit purposes of the AWS account. Which provides a record of actions taken by a user, role, or an AWS service in Cloud watch. It captures both the actions done in the AWS console and through CLI. Cloud Trail helps in troubleshooting also helps analyze with respect to Security.
AWS Config is an inventory of your resources in AWS. AWS Config also shows the history of the changes done to the resources with respect to Configuration. Also with AWS config we can define rules to check the configurations for compliance. This helps us to monitor our configurations at ease and could be used as continuous compliance.
Amazon S3 Storage is an Object Oriented Storage which is scalable, secure and durable. This is kind of platform less storage. Which has the capability to deliver 99.99% durability also Amazon S3 is used for write once and read many concept.
Amazon S3 has capability of getting 20000 Get Requests and 2000 Put Requests.
Amazon S3 has 5Gb of Standard Storage.
EFS is a storage service for Amazon EC2 instances. EFS is Scalable and Shared File storage service.
EBS (Elastic Block Store):
There will be situations where our application needs random IO, and more over the simple storage (storage that comes with the instance) is volatile that means they do not have durability – that means if the Instance is shutdown or rebooted the data stored in that storage will be deleted.
So this is the volume that gives us ability to do random IO and this will not just delete itself if we lose our EC2 instance.
That means the good thing is the Data or the device itself is independent from the Instance.
Amazon Glacier is a Long-term storage. It is like write once and read rarely.
Storage Gateway is a Hybrid Cloud Storage with seamless local integration. The Good Thing in this is there are NO Transfer Charges into AWS.
EBS: Amazon EBS (Elastic Block Storage)has persistent Block Level Storage volumes which is used with Amazon EC2 instances. EBS volume is replicated within its Availability Zone to protect from failures.
Difference between S3, EBS and EFS
|Called the Simple Storage Service which is Object Oriented.
S3 is Not just for EC2 but can be used outside EC2 Also. Files with in S3 bucket can be retrieved using HTTP protocols.
Data accessible from All regions.
S3 Typically experiences write Delays. Best suitable for write once read many.
5Gb of Standard Storage
20,000 Get Requests and 2,000 Put Requests
|Called the Elastic Block Store which is Block storage.
EBS volumes is limited to EC2.
Data accessibility is with in the region.
EBS does not have any Write Delays.
30Gb of EBS of any combination SSD or Magnetic.
2,000,000 I/O (with Magnetic)
And 1Gb of Snapshot storage
|Called The Elastic File System. Which is Object Oriented.
Can be attached to more then one EC2 machine. But is more expensive then EBS.
Accessible from Multiple EC2 Instances.
Faster then S3 but Slower then EBS.
This best suits for application oriented loads.
AWS Identity and Access Management:
This service isused to authenticate and authorize our users with in AWS. Authentication means to know who that person is and authorization means what that person is allowed to do. In this service we have ability to enable multifactor authentication. Multifactor authentication means you have UID and Password, also you have MFA device or soft token (Google authenticator, MS authenticator, etc. …) Which could be picked form your phone, which is an additional layer of security.
AWS Users & Groups: These are the general users and groups created to manage for AWS console.
Lab: Creating Users and Groups
From the services go to IAM.
Select Users from the left panel and click on add users.
Give the user name of your choice. We are creating “i2tutorialsuser” and select the Access Type. In this example we have selected “AWS Management Access Console”. This is to give the user access to “aws.amazon.com”
Where as “Programmatic Access” is for developers, which gives access to AWS CLI.
Click on Next Permission.
This will ask you to put the user in a group, however it also gives an option to create a group if you have not created one. Which is given below.
We already have few Groups however we have chosen to create a new group. We clicked on “Create Group” in the above screen. And then it is asking for Policies to attach it to the group. We have selected AWS managed Policies, and from them we have selected “Amazon S3 Full Access”.
We have selected the Policy and assigned a Group name. Then click on Create Group.
It will take us to the same screen of assigning a group to the user. Now select the Group which you created and proceed with “Next Review” followed by acknowledgement and then “Close”.
This is one way of creating group, however we could first create Groups and then create uses then assign them to the groups. In order to create the group first, just select Groups from the Left Panel and proceed with the same set of selection (Group name and Policy)
Access Keys: Usually we login to “aws.amazon.com” (we call this as AWS Management Console) using username and password. Using AWS access keys, we will have access to AWS CLI Commands.
Permissions and Policies:
We need to have a user account, but that user account by default does not allow the user to do any thing. In order for the user to be able to do their job and create resources within amazon AWS, we need to grant them permission. We have to have an explicit allow in order being able to do their job, so we do that by the way of policies.
We have two different types of policies.
AWS Managed Policies: AWS Managed Policies are policies, which are created and maintained by AWS.
Customer Managed Policies: Customer Managed policies are the policies, which are created and managed by our-self.
Roles are important and a powerful feature in AWS. Roles are the other way of authentication
Federated Users: This is single sign-on authentication for more then one application/login’s. Unlike many companies uses Active Directory Accounts for all their application/intranet.
MFA Device: This is an additional layer of security, Amazon provided MFA Device like our RSA device. After you have put the user name and password. It will prompt for the MFA token, only then we will be able to login to the AWS Management Console. Also A part from MFA we can use Soft Tokens (Any Authenticator Eg: Google-Authenticator)
RDS (Relational Database Service): Relational Database service is a web service running in the Cloud. Runs a scalable Relational database in the cloud.
Introduction to A Dynamo DB and Models:
Amazon DynamoDB is NoSQL Data stores. Which uses a different concept than Relational database service.
NoSQL Data stores means; it does not use SQL language but typically has a different pattern/concept for storing Data.
The advantage of Amazon’s Dynamo DB then RDS is that it is highly available and fault tolerant.
All of the Data is written to SSD volumes, and the data is replicated to multiple availability zones.
Types of Data Models. –> We will be creating tables in Dynamo DB.
PRIMARY KEY ATTRIBUTES
Just remember that each table is independent of each other.
Tables are made up of items, and items are made up of attributes.
Please notice here that I have included the name of the column inside each of these fields, and each attribute will include its name in the total storage for that particular item.
There is No limit to the size of the table.
However there is a limit to the size of the ITEM, which is 400Kb.
Creating Dynamo DB Table:
From the AWS Services, select “DynamoDB” under Database. And from the Amazon DynamoDB Dashboard select “Create Table”.
Give the Table Name as required and Primary key (Please refer to the Components link for more information on Primary Key Values). I have used the default table setting, how ever this could be modified as needed by un-checking “Use Default Settings”.
Once the table is created, the table looks as below.
Scan and Query Operations (Lab)
Once the table is created we might have a huge DB in which we might need to filter table for easy retrievals. I have created few ITEMS by clicking the “Create Item” button. We could either scan or Query the table with the necessary keys.
I have done a Query, However for Query we need to know the exact Partition Key or the sort key and some times both. Query with the key or you could select “scan” and hit the “Search” Button.
Amazon Elastic Cache
Amazon Elastic ache is like RDS. However things like, cumulative patches, OS patches, backup’s etc..are all done for us automatically in the background.
Amazon Elastic Cache has two choices.
Redis: We have Redis, which is an in-memory database. It supports multiple data types. It is also backed to disk so we do have the ability to stop and recover if needed
Memcache: Memcache is also an in-memory simple key value store. But is not backed to disk so we don’t have the ability to recover.
If there is complete loss of the Memcache cluster, then we will not have the ability to restore.
So this supports Clustering. i.e if one goes down the other is still in working condition.
Mobile and Application Services (Application Integration)
SQS (Simple Queue Service)
Amazon Simple Queue Service is a service that enables you to scale and differentiate services of the application/middleware, and has ability to scale independently with out the knowledge of each other.
Amazon Simple Queue Service highly available and fault tolerant. SQS could be put up in messages and buffer them so we could pull them after a period of time.
SNS (Simple Notification Service)
SNS is a mobile notification service responsible for delivery of massages from Endpoints and Clients. This service is push-based service.