AWS

EC2 - Elastic Compute Cloud

Three instances types:
1. On Demand - Run as per requirement
2. Reserved - Reserved Upfront
3. Spot Instances - Bid for unused instances
Instances are configured for variety of purposes and are grouped based upon their applucation
like:
1. General Purpose Instances
2. Memory Optimised Instances
3. Computing Optimised Instances
4. GPU Optimised Instances
and the above EC2 instances are sub grouped in their own categories to provide a variety of
instances for the users based upon their needs.
Security Group - Security group defines a set of rules according to which the permission to
access the instances is granted (like a firewall).
It is based upon port and ip-mapping. It can be easily configured by the user based upon his
needs and wants to provide access to whom.
EBS - Elastic Block Storage
Three types:
1. General Purpose SSD - Now Default
2. Provisioned IOPS (PIOPS) - Provisioned by the user according to his needs
3. Magnetic - Older can be used to store less intensive data.
The performance of EBS is based upon two primary factors:
1. IOPS - Input/Output per second
2. Throughput - Read/Write rate to storage
All the GPS volumes comes with a predefined set of IOPS and also can burst @ 3000 IOPS for
an extended time. The IOPS are accumulated at 3 IOPS/GB. The max IOPS supported in this
system is 10,000 IOPS and 160MB/s of throughput.
Baseline performance for 100GB SSD: 100*3 IOPS/GB = 300 IOPS (for 100GB GPS)
When your volume requires more than the baseline performance I/O level, it simply uses I/O
credits in the credit balance to burst to the required performance level, up to a maximum of
3,000 IOPS.

With Provisioned IOPS (SSD) volumes, you can provision a specific level of I/O performance.
Provisioned IOPS (SSD) volumes support up to 20,000 IOPS and 320 MB/s of throughput. This
allows you to predictably scale to tens of thousands of IOPS per EC2 instance. The ratio of
IOPS provisioned to the volume size requested can be a maximum of 30; for example, a volume
with 3,000 IOPS must be at least 100 GiB.

EBS volumes behave like raw, unformatted block devices. You can create a file

system on top of these volumes, or use them in any other way you would use a block
device (like a hard drive).
You can use encrypted EBS volumes to meet a wide range of data-at-rest
encryption requirements for regulated/audited data and applications.
You can create point-in-time snapshots of EBS volumes, which are persisted to
Amazon S3. Snapshots protect data for long-term durability, and they can be used as
the starting point for new EBS volumes. The same snapshot can be used to instantiate
as many volumes as you wish. These snapshots can be copied across AWS regions.
EBS volumes are created in a specific Availability Zone, and can then be
attached to any instances in that same Availability Zone. To make a volume available
outside of the Availability Zone, you can create a snapshot and restore that snapshot to
a new volume anywhere in that region. You can copy snapshots to other regions and
then restore them to new volumes there, making it easier to leverage multiple AWS
regions for geographical expansion, data center migration, and disaster recovery.
Performance metrics, such as bandwidth, throughput, latency, and average
queue length, are available through the AWS Management Console. These metrics,
provided by Amazon CloudWatch, allow you to monitor the performance of your volumes
to make sure that you are providing enough performance for your applications without
paying for resources you don't need.
S3 - Simple Storage Service
Secure, durable, highly scalable storage service. It is based on write once read many approach.
Content storage and distribution, Disaster Recovery, Static website hosting etc.
Two components:
1. Buckets - Container Service for Objects, organises s3 at highest level of
hierarchy, provides access control, unit for usage reporting
2. Objects - Files, folders stored inside of the Buckets.

Objects consists of Data opaque to s3, MetaData which is set of name-value pair that describes
the object and Object is uniquely identified inside the bucket by a key (name) and a Version ID.
Keys: Used to uniquely identify objects inside buckets. Every Object inside a bucket has a
unique key.
A combination of bucket, key and version ID uniquely identify objects inside amazon s3.
Permissions can be set in 3 different ways in S3
1. User-Based Permissions/IAM (By creating/modifying users and attach roles to
them from AWS Management console from existing roles or creating new roles)
2. Bucket Policies (Permissions can be set on a bucket level so that the objects
inside the bucket will have those permissions, fine grained) [ex: grant all users read only
permission on buckets]
3. ACLs (Grant permission to bucket or its objects at object or bucket level, coarse
grained approach, can apply basic read/write permissions to other AWS accounts, limits
to grant permissions by using ACL)
Storage Classes:
1. Standard: Designed for high redundancy 11 9s percent. Designed to sustain loss
of data in two facilities (provides good data replication so high redundancy)
2. Reduced Redundant Storage: Its less redundant than standard stoarge class
also less expensive. Provides 99.99% durability.
3. Glacier: Based upon write once never read approach. Used for archiving data
where data is infrequent and sereval hours of retrival time is acceptable.
Storage class in S3 storage can be modified as required or while objects are loaded.

Other features of S3:

Encryption: S3 by default does encryption (SSE-SE) of data using its standard encryption
library (256 bit symmetric key). Provides a good level of encryption using multiple keys.
a. Data is encrypted by a key.
b. The key used to encrypt the data is itself encrypted using a master key.
c. The master key is itself rotated on a regular basis for security.

User can also encrypt its data using his own key(SSE-C). This key is not stored by S3 and is
immediately discarded after encryption is done.
User can use AWS key management serivce for encryption of data (SSE-KMS). This approach
also provides auditing features where one can see who has used his encryption key and when.
Audit Logs: Provides logging facility for auditing purposes.
Multi-Factor Authentication Delete
Time limited access to objects
Versioning: Select the object and click on versioning. To have multiple versions of the same
object.
Cross region replication of Data: create new bucket in a different region. Activate replication in
the source bucket and point it to the destination bucket in the other region.
Lifecycle Rules (Archive to Glacier): Lifecycle of objects can be configured in which after the
defined duration the object will get archived in Glacier and after the defined permanent delete
duration the object will be deleted from the Glacier as well. Retrieving of objects from glacier
takes time.
Static Website Hosting: Can be used to host static website using buckets and one can also use
Route53 to manage DNS service. Scales automatically according to traffic demands. [ex: create
two buckets named site.com (with default index.html and error.html page) www.site.com bucket
which redirects to the site.com bucket. Set A record of website to point to site.com bucket and
CNAME WWW to www.site.com bucket.]
Regions and Availability Zones
Regions are the Geographical locations where one can configure different AWS services ex:
Sydney, Singapore, Tokyo etc.
A Region can have multiple Availability zones. AWS allows us to place resources in multiple
locations.
Each region is completely independent. Each Availability Zone is isolated, but the Availability
Zones in a region are connected through low-latency links.
Amazon EC2 resources are either global, tied to a region, or tied to an Availability Zone.
Regions: Each Amazon EC2 region is designed to be completely isolated from the other
Amazon EC2 regions. This achieves the greatest possible fault tolerance and stability.

Amazon EC2 provides multiple regions so that you can launch Amazon EC2 instances in
locations that meet your requirements.
When you view your resources, you'll only see the resources tied to the region you've specified.
This is because regions are isolated from each other, and we don't replicate resources across
regions automatically.
Availability Zones: When you launch an instance, you can select an Availability Zone or let us
choose one for you. If you distribute your instances across multiple Availability Zones and one
instance fails, you can design your application so that an instance in another Availability Zone
can handle requests.
Note: The various resources are tied to different levels in AWS. As mentioned:

Resource
AWS

Type

Description

Global

You can use the same AWS account in all regions.

Global or

You can use the key pairs that you create using Amazon EC2

Regional

only in the region where you created them. You can create and

Account
Key Pairs

upload an RSA key pair that you can use in all regions.
For more information, see Amazon EC2 Key Pairs.
Amazon

Regional

Each resource identifier, such as an AMI ID, instance ID, EBS

EC2

volume ID, or EBS snapshot ID, is tied to its region and can be

Resource

used only in the region where you created the resource.

Identifiers
User-

Regional

Each resource name, such as a security group name or key

Supplied

pair name, is tied to its region and can be used only in the

Resource

region where you created the resource. Although you can

Names

create resources with the same name in multiple regions, they

aren't related to each other.

AMIs

Regional

An AMI is tied to the region where its files are located within

Amazon S3. You can copy an AMI from one region to another.
For more information, see Copying an AMI.
Elastic IP

Regional

Addresses
Security

An Elastic IP address is tied to a region and can be associated

only with an instance in the same region.

Regional

Groups

A security group is tied to a region and can be assigned only

to instances in the same region. You can't enable an instance
to communicate with an instance outside its region using
security group rules. Traffic from an instance in another region
is seen as WAN bandwidth.

EBS

Regional

Snapshots

An EBS snapshot is tied to its region and can only be used to

create volumes in the same region. You can copy a snapshot
from one region to another. For more information, seeCopying
an Amazon EBS Snapshot.

EBS

Availability

An Amazon EBS volume is tied to its Availability Zone and can

Volumes

Zone

be attached only to instances in the same Availability Zone.

Instances

Availability

An instance is tied to the Availability Zones in which you

Zone

launched it. However, note that its instance ID is tied to the

region.

VPC
Used to create own private cloud inside AWS and configure networking policies just like in local
data center using Network ACLs, Security Groups, NAT and Network Gateways, VPNs.
By default now AWS comes with a default VPC preconfigured which can be used or user can
create their own VPC.
It has two components:
1. VPC
2. Subnets
VPC belongs to a particular region ex: 10.0.0.0/16 is a VPC and it can have multiple Subnets
present in different availability zones. Here by basic networking we can have 256 subnets of the
type 10.0.1.0/24, 10.0.2.0/24,10.0.3.0/24 and so on

These subnets can belong to a single az or can be present in multiple az. Instances can be
booted in multiple subnets present in different az to achieve high availability of resources.
By default All subnets within a VPC can communicate with each other using private ip
addresses.
Instances inside a Subnet can be connected using ssh in the following manner:
1. Using VPN tunnels
2. Assign gateway to VPC (only one gateway can be assigned to each VPC),
Assign public ip to the instance, configure route so that traffic can pass through the
internet gateway. then ssh is possible to public ip address.
3. The NAT is used in case only if instances wants do download packages,
softwares and updates from the internet. This cannot be used in connecting us with the
instances.
The VPC in Amazon can be connected to our existing infrastructure securely using IPSec VPN
connection. We can also configure it for HA using Direct Connect and VPN for failover
conditions. For more higher HA each subnet can have a separate vpn connection using BGP (2
IpSec tunnels). A total of 4 Ipsec tunnels and multiple gateways on customer ends enhances the
availability further. Using Aws direct connect in the approach with VPN as backup eliminates the
possible chances of downtime.
In VPC we can have one Subnet publicly available through internet gateway and other subnets
as private.
We can configure software VPN for VPC to VPC connectivity.
Software firewall to internet can be done using NAT instances and configuring routes so that all
traffic to internet passes through these instances.
VPC Peering: It enables one VPC to communicate with other VPCs. It is a really helpful
approach which can be architectured for clients based upon their requirements. ex: single VPC
in center communicating with other VPCs to gather logs from multiple VPCs or for code
deployment strategies.
VPC Peering can also be done across accounts.
Note:

(Transit connection is not possible ex: connecting VPC C from VPC A via VPC B).
Also, peer VPC address ranges cannot overlap.
Security groups not supported across VPCs
Nat can also be configured for VPC peering in which traffic will flow to and fro between the VPC
through the NAT instances.
AWS Elastic Load Balancer
Used to balance incoming traffic load among the instances
In AWS VPC ELB is supported both on public ip address and on private ip addresses.
Full control over ELB Security group.
The ELB is created on a separate VPC from that of the servers and the interface is securely
connected with the servers. If its a private ELB a private ip-address is assigned if public then a
public ip address is assigned.
Two types of listeners
1. TCP/SSL
Incoming client connection bound to server connection, no header is attached from elb,
round robin algo used.
2. HTTP/HTTPS
Incoming client connection terminates at ELB and pooled to the server, headers may be
modified, uses x-forwarded for header for client ip-address, least outstanding request
algo used for routing, sticky session support session available.
Health Checks
Two types of health checks:
1. TCP (not that reliable, ICMP ping response)
2. HTTP (more reliable, must return a 2xx response)
Idle Timeouts
Length of time for which client connection should be kept open. Default is 60 seconds can be
set between 1 to 3600 seconds.
It allows the ELB to close the client session when no longer in use.

We can have Multiple instances running the same application in different availability zones and
also multiple ELBs in different availability zones pointing to the corresponding instances and
Route53 load balancing incoming traffic on the different load balancers.
We can also have a situation in which we have instance/s in a single availability zone and ELBs
in multiple availability zones pointing to the same instance/s present in the single availability
zone.
By default now it is recommended to have the ELB present in multiple availability zones to
increase redundancy.
Cross Zone Load Balancing
Using cross zone load can cause traffic in balance between the instances to resolve this it is
better to enable cross zone load balancing among the elbs.
SSL and HTTPS ciphers are used according to industry standards.
Queue of upto 1024 is supported by load balancer in case the instances are not able to scale to
such an extent after which a 503 error is shown to the client if connection exceeds.
elb also supports logging which records client ip, date , time, request path and server response.
elb can handle failure of single availability zone, Route53 re routes the traffic away from the
failed elb.
RDS
Database services provided by AWS. Consist of Mysql, SQL, Postgres and Oracle databases.
No need to manage database (backups/update/upgrade etc.) done by AWS. Just upload the
data to the database after spinning an RDS.
No need to install any software or DB software done by AWS itself.
We can replicate the database to multiple availability zones to ensure high availability in case of
any failures.
Also supports cross region replication.
Data File upload can be done in two ways

1. Standard Mysql: Processes sequentially, restart if failed, good for small

databases.
2. Flat file: Load schema initially then the data, resume data restore in case of
failure, best for large databases.
Can also be configured for Replication. Data can be warehoused in AWS Redshift from RDS as
a source.
Also supports read replicas which only serve read requests of the client. Scale the RDS
instance by considering Read and Write Ratios and using multi read replicas if application is
more read oriented and distribute the read traffic among the replicas (note: this will not affect
the write operation as data will have to written on all replicas equally).
InnoDB Buffer warming: Dump buffer pool, reload after reboot is complete. Cache warmimg can
be done by defining the time period (like 1 hour) so that if a failure occurs after half an hour than
we will get cache warmed up though it will be 30 minutes stale (performance improvement than
may not be high).
Changing schema: can be done in multiple ways
1. Using Read replica: promote the read replica to R/W and make the schema
changes in it than create a read replica from it and promote the R/W replica as
master(Not easy, may break replication).
2. Native 5.5 feature: Easy to use, less performance impact. Some performance
implication. Takes relatively less time.
3. PT Online schema change: run from an ec2 instance to perform schema
changes. Needs installation, could take longer, less performance impact.
Burst Mode: GP2 and T2
GP2 uses SSD and gives 3IOPS/GB base and 3000+ IOPS on burst
T2 gives credits per hour when below base performance, can store 24 hours of credits AWS
cloudwatch matrix to see credits.
CloudFront
Caching facility provided by AWS which transmits copy of our content to various edge locations
so the users are served the content with less latency and reduces load on base instances.
Content is divided into 3 categories:

1. Static Content (images, css, javascript etc., can be distrubted to more than one
user, doesnt change for sec, min, hours)
2. Dynamic Content (php, html etc.)
3. Live Streaming (videos, live streaming etc.)
Static Content on S3
Transfer of static content from S3 to Cloudfront is free, decrease load on server, highly available
and scalable
Control access to content on S3 using OAI (Origin Access Identity), content can be accessed
only via cloudfront, ensures content is not leaking and S3 url is not being used anywhere else.
Control Access to content on Cloudfront using Signed Url (ex: marketing email) or Signed
Cookies (ex: streaming, whole site authentication).
Cache at every layer: browser caching. Set max-age or expiry date in headers, HTML5
application cache, helps eliminate network latency, browser cache is limited
Cache at every layer: edge caching. Set High TTL for intermediary caches, dont forward query
strings or headers or cookies. Use cloudfront defaults.
Version Objects
Allows for easy updates and rollback, use version names so no additional api calls are needed,
set high TTL on objects that do not change frequently, each object is treated as a unique object
in the browser
Dynamic Content
Content unique to every request(/index.php), Content Changes frequently (seconds, minutes)
but not unique for every request (Weather updates, API etc.), content changes on end user
request (query strings, cookies, headers) (ex: mobile vs desktop users, search keyword in
query strings etc.)
Cloudfront supports TTL as low as 0 seconds. no-cache, no-store etc. most content can be
cached even if for few seconds, helps offload origin load, cloudfront will server stale content if
origin is unavailable, supports if-modified-since, if-none-match when object in cache has
expired.
Provides reporting for various objects.
Multiple cache behaviour. Only forward required headers. avoid forwarding user agent header,
avoid forwarding all cookies

Streaming Media
Live streaming, On demand streaming, audio streaming, generally involves transmit of manifest
file, media file and player. Manifest file (TTL: 2sec), Media file (60sec), Player (static store in S3,
high TTL 24 hrs)
Use HTTP based streaming protocols, use cloudfront with ec2 for media straming,use
fragmented streaming formats(native in cloudfront), HLS etc, dont forward header, query string,
cookies, use signed cookies.
We can set custom error pages in cloudfront.
Design for failure. Set Route 53 to point to two elbs instead of one so in case of failure the
cache can get data from servers in separate availability zones.
Supports SSL between browser and edge also between edge and origin, redirect HTTP to https
to use encryption in apache config.
Use AWS IAM (for users to permit access to APIs)< Use Cloudtrail to record API calls. Use
*.cloudfront.net SSL. Certificate cost per IAM certificate, Can use single UCC certificate for
multiple two level domains, can also use wildcard certificate for three level domains.
Use different price classes for various edge locations and optimize cost accordingly.
Domain Sharding. Many browsers limit the number of parallel connections to a single domain,
shard assets over multiple domains, Use multiple CNAME aliases and wildcard Cert for SSL.
Also supports Access log for analysis.
Route 53
Route 53 not only can be used for DNS resolving but also for HA of our resources in case of any
failures at the DNS level itself.
Route 53 can be configured for automatic failover in which case if a failover occurs it reduces
the downtime with a considerable amount of time and takes actions automatically without
human intervention.
Route 53 checks for two parameters:
1. Health: Is the ELB or EC2 is available or not
2. Weight: How much weight is present on each ELB or EC2 in multiple subnets.
ex: If we have two elbs serving the same content with a Weight of 50 on each and suppose one
of the elb goes down then the load will be shifted by route53 to the other elb ie. to 100 which
increases availability of application in case of any failure.

If the other elb also due to any reason goes down then the Route53 does health checks and if
found negative switches to the other failure process (S3 bucket content in this example)
automatically.
ex:
Record

Type

reinv.net
reinv.net

Health Check

Value

Weight

ALIAS (Evaluate Target)

prod.reinv.net

100

ALIAS

reinv-fail.s3-web

prod.reinv.net ALIAS (Evaluate Target)

prod-1.elb

50

prod.reinv.net ALIAS (Evaluate Target)

prod-2.elb

50

Glacier
Amazon Glacier is an extremely low-cost storage service that provides durable storage with
security features for data archiving and backup. With Amazon Glacier, customers can store their
data cost effectively for months, years, or even decades. Amazon Glacier enables customers to
offload the administrative burdens of operating and scaling storage to AWS, so they don't have
to worry about capacity planning, hardware provisioning, data replication, hardware failure
detection and recovery, or time-consuming hardware migrations.
Amazon Glacier is a great storage choice when low storage cost is paramount, your data is
rarely retrieved, and retrieval latency of several hours is acceptable.
Amazon Simple Storage Service (Amazon S3) supports lifecycle configuration on a bucket that
enables you to optionally transition objects to Amazon Glacier for archival purposes.
In Amazon Glacier, a vault is a container for storing archives, and an archive is any object, such
as a photo, video, or document that you store in a vault.
Any archive operation, such as upload, download, or deletion, requires that you use the AWS
Command Line Interface (CLI) or write code. There is no console support for archive operations.
For example, to upload data, such as photos, videos, and other documents, you must either use
the AWS CLI or write code to make requests, using either the REST API directly or by using the
AWS SDKs.
aws-cli to upload archive to glacier:
aws glacier upload-archive --account-id - --vault-name my-vault --body archive.zip
In general, retrieving your data from Amazon Glacier is a two-step process:
1. Initiate a retrieval job.
2. After the job completes, download the bytes of data.

The retrieval job executes asynchronously. When you initiate a retrieval job, Amazon Glacier
creates a job and returns a job ID in the response. When Amazon Glacier completes the job,
you can get the job output (archive data or inventory data).
Note:
1. Files uploaded to glacier vault can also be uploaded in chunks which enables
better performance and resume capability if the internet connection is not stable.
2. Glacier also provides granular security and compliance in which users can
configure strict policies ex: any archive stored in amazon vault must have to be retained
for a duration of 365 days and cannot be deleted even by super user account (admin).
Amazon allows to review these policies in a period of 24hrs because once a policy gets
applied it cannot be rolled back afterwards. These policies can be easily configured and
associated with vaults using AWS console.
DynamoDB
Amazon DynamoDB is a fully managed NoSQL database service that provides fast and
predictable performance with seamless scalability. No need for administrative tasks like scaling,
software updates, patching etc. done at AWS level itself.
With DynamoDB, you can create database tables that can store and retrieve any amount of
data, and serve any level of request traffic. You can scale up or scale down your tables'
throughput capacity without downtime or performance degradation, and use the AWS
Management Console to monitor resource utilization and performance metrics.
Tables
Similar to other database management systems, DynamoDB stores data in tables. A table is a
collection of data.
Items
Each table contains multiple items. An item is a group of attributes that is uniquely identifiable
among all of the other items.
Attributes
Each item is composed of one or more attributes. An attribute is a fundamental data element,
something that does not need to be broken down any further.
Primary Key

When you create a table, in addition to the table name, you must specify the primary key of the
table. As in other databases, a primary key in DynamoDB uniquely identifies each item in the
table, so that no two items can have the same key.
RedShift (MPP Database for high vol structured data)
Redshift is data warehousing service provided by amazon. It is used for databases that are
highly structured but also very large in size which cannot be handled by traditional relational
databases which are best suited for high structured data but able to manage low quantity of it.
Redshift can be easily scaled from a 2TB datastore to many PB (PetaBytes) of data storage.
Amazon Redshift comprises of a single master node and adjacent computing nodes. The
queries which comes up through master node is equally distributed among the computing nodes
which run the queries in a parallel fashion and returns the end result. This results in an overall
reduction in execution time of the query as compared to a single node executing it. The same
holds true for writing in these kind of databases. Works with existing analysis tools.
Features:
1. Standard SQL
2. Optimized for fast analysis
3. Very scalable
Redshift Features:
1. Uses column based approach to store data (row based storage causes additional
I/O, with column approach only read data that we need)
2. Uses Data Compression (COPY compresses automatically, can analyze and
override, more performance less cost)
3. Zone mapping (Track minimum, maximum value for each block, skip over blocks
that dont contain relevant data)
Data can be loaded into Redshift using S3, DynamoDB, EMR, Linux box. If already have data
sources from which you want to upload data to Redshift either frequently or infrequently use a
intermediate layer like ETL(Execute, Translate, Load) service provided by many third parties
like: informatica etc.
EMR (Elastic Map Reduce) (Hadoop)
EMR is the service for Applications that require Hadoop infrastructure. This kind of databases
are best suited for data which is large in size and also highly unstructured. Hadoop uses a
clustering approach in computing. It includes:

1. Input file: This is the data on which computing has to be performed. Partitioned
into multiple files.
2. Functions: The operation that has to be performed on the data.
Putting Data into EMR
1. Put the data into S3
2. Choose hadoop distribution, no. of nodes, types of nodes, hadoop apps live hipe,
pig etc.
3. Launch the cluster using EMR console. CLI, SDK or APIs
4. Get the output from S3
By adding data into S3 it can then be used by many EMR clusters for analysis on the same set
of data running different functions and output the result back into S3.
The clusters can be easily resized and many parallel clusters can be executed in AWS.
Can also use spot instances for Hadoop computing nodes. The cluster can also be terminated
after the analysis is complete to reduce costs.
The data can also be stored in HDFS (local disk) if you dont want to store it in S3.
EMR Cluster is divided into 3 parts:
1. Master instance group: It is the primary managing node in the hadoop cluster.
2. Core Instance group: These are the instances that have HDFS(local disk)
attached with them and performs processing on the data to provide output.
3. Task Instance group: This instance group doesnt has any HDFS attached with
them. These are only used for computing on the data. This instance group can also be
provisioned with spot instances to reduce cost. Multiple Task instance groups can be
attached in a single EMR cluster and can be provisioned with spot instances for multiple
price ranges. By using multiple spot options in this group helps in managing cost and
also managing fluctuations in processing power if spot instances become over priced.
Lambda
AWS Lambda is a compute service where you can upload your code to AWS Lambda and the
service can run the code on your behalf using AWS infrastructure. After you upload your code
and create what we call a Lambda function, AWS Lambda takes care of provisioning and
managing the servers that you use to run the code.
AWS Lambda runs your code on a high-availability compute infrastructure and performs all of
the administration of the compute resources, including server and operating system
maintenance, capacity provisioning and automatic scaling, code monitoring and logging. All you

need to do is supply your code in one of the languages that AWS Lambda supports (currently
Node.js, Java, and Python).
When using AWS Lambda, you are responsible only for your code. AWS Lambda manages the
compute fleet that offers a balance of memory, CPU, network, and other resources. This is in
exchange for flexibility, which means you cannot log in to compute instances, or customize the
operating system or language runtime. These constraints enable AWS Lambda to perform
operational and administrative activities on your behalf, including provisioning capacity,
monitoring fleet health, applying security patches, deploying your code, running a web service
front end, and monitoring and logging your Lambda functions.
The lambda function can be configured as
1. event driven: If a particular event occurs the lambda function will be called.
2. HTTP request: The lambda function will be executed if an HTTP request is made
using the APIs.
Lambda can be used to create backends, event handlers and data processing systems.
Services supported by lambda includes: S3, DynamoDB, Kinesis, Cloudformation, SWF,
Cloudtrail, SES, Cognito, API Gateway, SNS, Cloudwatch logs.
(ex: lambda can be used for auditing cloudwatch logs, where cloudwatch calls the lambda
function which in turn can process the requests and store the results in DynamoDB, S3, or
Redshift. ex: Lambda can also be used to create thumbnail of images being uploaded to S3
bucket.)
Lambda can now run functions for up to 5 minutes.
It also now supports versioning capability where we can use different versions of the same
code, aliases can also be associated to these different versions of the code.
Resource sizing can be done in lambda to provision the resources that will be used while
running the lambda functions or it can be left to be done by lambda itself automatically.
Lambda also supports scheduled execution of lambda functions using standard cron syntax.
Lambda functions can also be provisioned to run inside our own private VPC.
IAM
Identity Access management provides a means by which we can control which users can
access what resources inside of AWS. It basically is divided into categories like:
1. User: Defines the user and its corresponding credentials (password, Multi-factor
authentication, access key)
2. Groups: This comprises of group of users with a predefined set of access
permissions to AWS resources.

3. IAM Policies: These policies can be attached with users or groups and defines
the resources and the level of permission on those resources and whether the access is
allowed or not.
Best practices:
1. Grant least privileges to users: Grant only the permissions that are required.
2. Is users will be accessing the resources only through code and api requests then
allow access using access keys if they require access through management console
then only grant them access through passwords.
3. Define users in groups and assign permissions to the group this helps in easy
management.
4. Restrict access by adding conditions.
5. Enable AWS Cloudtrail to get logs of all the API calls being made.
6. Rotate security credentials regularly.
7. Use multi factor authentication.
8. Use IAM roles to share access.
9. Use IAM roles on ec2 instances to access resources.
Code Commit
AWS CodeCommit is a fully-managed source control service that makes it easy for companies
to host secure and highly scalable private Git repositories. CodeCommit eliminates the need to
operate your own source control system or worry about scaling its infrastructure. You can use
CodeCommit to securely store anything from source code to binaries, and it works seamlessly
with your existing Git tools.
Access to these repositories can be given to users using IAM.
Code Deploy
Code Deploy automates the manual process of deploying code to a single or multiple instances.
It needs codedeploy agent running on ec2 instances to be able to use it. It can also be used to
deploy code at on-premises servers or at any other data center which is chargeable but not
chargeable if used for ec2 instances. Code Deploy can be configured with Git repositories and
adding Appspec YAML file with the application code. The YAML file consists of three important
details:
1. file source and destination: gives the source and destination location to which the
files will be copied.
2. File permissions: can be used to grant privileges to different files/directories in
the code.

3. Hooks: These are different kind of hooks which can be executed at different
levels. Ex: Setup of apache server, starting apache server, restarting apache server.
These hooks can execute different kinds of scripts like sh, python etc. and perform
different kind of operations we configure in those scripts.
CodeDeploy helps in completely automating the process of deployment based upon different
approaches:
1. One at a time: In this approach one server is updated at a time with rest others
serving the users.
2. Half at once: Half of the instances are updated once leaving the rest for
delivering content.
3. All at once: All the servers are updated at once.
4. Custom: Define our own number of instances for deployment at a time.
This helps in updating code running on multiple servers easily and effectively. If code deploy
finds any abnormality in the updation process it automatically rolls back to previous edition of
the code. It detects updation process as unsuccessful if the user defined scripts generates error
during their execution or flags as successful if all the scripts defined by the user executes
successfully. Code Deploy can also be configured with integration tools like Jenkins etc.
Code Pipeline
AWS CodePipeline is a continuous delivery service for fast and reliable application updates.
CodePipeline builds, tests, and deploys your code every time there is a code change, based on
the release process models you define. This enables you to rapidly and reliably deliver features
and updates. You can easily build out an end-to-end solution by using our pre-built plugins for
popular third-party services like GitHub or integrating your own custom plugins into any stage of
your release process. With AWS CodePipeline, you only pay for what you use. There are no
upfront fees or long-term commitments.
You will be able to use the CodePipeline user interface to construct a graphical model of your
release process using a combination of serial and parallel actions. The workflow can include
time-based or manual approval gates between each stage. For example, you could choose to
deploy new changes to staging servers only during weekday working hours in your own time
zone.
CodePipeline watches your source code repo for changes and triggers the appropriate
workflow. A release workflow could build the code in a production build tree, run test cases, and

deploy tested code to a staging server. Upon final approval (a manual gate), the code can be
promoted to production and widely deployed.
Code Pipeline has variety of stages in configuration phase:
1. Name of the pipeline: Give any name as required.
2. Source: Code Source (Amazon S3, Github)
3. Build Provider: Choose if project requires creation of build (ex: for java based
projects). Two options are present: a. No Build, b. Jenkins (give jenkins url, name of
project in jenkins)
4. Beta: This is the beta or testing stage in pipeline (This acts as the staging phase
in the pipeline, production stage will be added later on after the pipeline is built using the
initial UI). We can define how we want to deploy the code in the Beta phase either by
CodeDeploy or by Elastic Beanstalk. Ex: Selecting CodeDeploy define the Application
Name present in CodeDeploy as well Deployment Group present in CodeDeploy which
will handle the Beta release.
5. Service Role: This will redirect to IAM roles. Define the role which will be used by
CodePipeline and give it permission of Allow so that code pipeline gets permission to
deploy the code.
6. Review: This stage gives an overview of all the settings that the user has defined
in his pipeline.
After the above steps have been completed we are provided with a graphical representation of
our pipeline with stages such as: a. Source (Representing Github) and Beta (Defining
CodeDeploy for Code Deployment). Now we can add more stages in this pipeline using the UI
and define the actions which will take place inside of that stage. In this manner we can create
our own custom pipeline which defines our complete release process with multiple stages for
staging, testing and production phases as per our requirement.
CloudFormation
AWS CloudFormation gives developers and systems administrators an easy way to create and
manage a collection of related AWS resources, provisioning and updating them in an orderly
and predictable fashion. You can use AWS CloudFormation sample templates or create your
own templates to describe the AWS resources, and any associated dependencies or runtime
parameters, required to run your application.

You can also visualize your templates as diagrams and edit them using a drag-and-drop
interface with the AWS CloudFormation Designer.
The CloudFormation has following components.
1. Template: Templates consists of a group of resources with their properties which
will build up your application infrastructure. Templates can be configured using GUI tool
known as CloudFormation Builder. The format of template is json. Each resource is
listed separately and specifies the properties that are necessary for creating that
particular resource.
2. Stack: Before you create a stack from a template, you must ensure that all
dependent resources that the template requires are available. A template can use or
refer to both existing AWS resources and resources declared in the template itself. AWS
CloudFormation takes care of checking references to resources in the template and also
checks references to existing resources to ensure that they exist in the region where you
are creating the stack. If your template refers to a dependent resource that does not
exist, stack creation fails.
3. Builder: Graphical Interface provided by AWS which can be used to create
Templates by visualising different resources with their relations and dependencies.
Cloudformation also supports capability to check the progress of stack creation the different
resources created during stack formation are also labelled accordingly to make recognition
better.
The resources created during the creation of stack can be cleaned up easily by deleting the
stack.
Elastic BeanStalk
Aws service by which we can deploy different kinds of applications (ex: python, node.js, php,
java etc.) from simple clicks of a button and define what infrastructure resources are needed for
the application with their respective properties without the need for us to configure those
resources. ex: suppose we have a compressed python application in the form of zip file, By
using Elastic BeanStalk wizard we can:
1. Define the name of the beanstalk with a short description
2. Now set the environment tier i.e. Web Server, Select the platform i.e. Python for
our example, environment type i.e. Autoscaled or Single Instance.
3. upload the compressed form of the application or upload it into s3 and provide its
url we can also use sample application.

4. Environment information: Name your environment or go by default. Also consists

of the environment url and provide description for the environment if required.
5. Configure RDS properties if your application requires database connectivity, we
can also configure vpc in this wizard.
6. In the Configuration details page select the type of instance you want for your
application, key pair to be used, email address, root volume type etc.
7. Environment tags can be configured to tag our environment resources.
8. Review the configured settings and Launch if the settings meets up our
requirements.
It also does health check of our application and in case of any error the beanstalk process rolls
back. we can now access our application with the elastic beanstalk url provided to us by the
interface.
Application updates can also be performed using eb cli to using eb deploy after configuring
some setting parameters for elastic beanstalk cli.
It has 3 components:
1. Environment: Infrastructure resources like EC2, elb, autoscaling, runs single
application version at a time, an application can have multiple environments like testing
and production.
2. Application versions: Application code, Stored in S3, An application can have
many application versions (easy to rollback to previous versions)
3. Saved configurations: Configuration that defines how an environment and its
resources behave, can be used to launch new environment quickly or roll back
configuration, an application can have multiple saved configurations
Beanstalk also provides us with a temporary CNAME that can be used for application testing
purposes.
Application Versions: An application can have multiple versions stored in S3, code can also be
deployed from github
Application Environments: An application can have multiple environments like testing and
production and a particular version of the application can be deployed in the environment
Saved Configurations: Save these for easy duplication for A/B testing or non-desruptive
deployments.
Deployment can be done either by using aws management console, cli or eclipse or VS add ons
Initial App Workflow using cli:

1.
2.
3.
4.
5.

Initialize git repo: git init .

Create elastic beanstalk app: eb init, and follow prompt to configure environment.
Add your code: git add .
Commit: git commit -m message
launch application using beanstalk: eb create

Update your app:

1. Update code.
2. Push updated code: git add . , git commit -m v2, eb deploy
3. Monitor Deployment status: eb status
Add extensions to your environment using ebextensions like sqs, cloudfront.
Zero Downtime Deployment: Using rolling deployments to achieve zero downtime deployments
using beanstalk.
OpsWorks
OpsWorks enables us to create our infrastructure stack using the capabilities of chef and its
recipes. Opsworks provides us with more control over our infrastructure than provided by
Beanstalk (which is more suitable to deploy a simple web application, and all the configuration
part is handled by amazon itself). Opsworks is based upon layered architecture where different
resources like servers, databases etc. are added as different layers inside the stack. By using
opswork we can not only define our the layers (instances, database etc.) which will be part of
our infrastructure stack but we can also customize these layers as per our needs. We can also
execute sample chef recipes provided by amazon or can also create our own recipes which will
be executed using opswork. Opsworks can also be used for on premises infrastructure. It also
easily can be integrated with third party CGI tools for application deployment using tools like
Jenkins, Travis etc.
1. Create a Stack. A stack contains the set of Amazon EC2 instances and instance
blueprints, called layers, used to launch and manage these instances (e.g., all the PHP
servers and the MySQL database used for your production web application). Apps, user
permissions, and resources are scoped and controlled in the context of the stack.
2. Define the layers of your stack. A layer defines how to set up and configure a set
of instances and related resources such as volumes and Elastic IP addresses. AWS
OpsWorks includes layers for common technologies such as Ruby, PHP, HAProxy,
Memcached, and MySQL (Linux only), and makes it easy to extend existing layers or
create custom layers. Lifecycle events can be used to trigger Chef recipes on each

instance to perform specific configuration tasks. For example, the deploy event could
trigger a script to create a database table for a new app.
3. Assign instances to your layers. Create instances in configurations you choose,
including instance size, Availability Zone, volume creation and RAID configuration, EIP,
security group, and operating system. Start your instances, or apply them to auto scaling
groups.
4. Define and deploy your apps. To define an app, simply tell AWS OpsWorks where
it can find your code and specify any additional deployment tasks, such as database
configuration. AWS OpsWorks supports a variety of repositories such as Git, SVN,
HTTP, and Amazon S3. When you deploy the app, AWS OpsWorks will pull code from
your repository, place it on the instances, and run the specified deployment tasks so that
your application is properly configured. Afterwards, you can view your apps deployment
logs to review the deployment steps, verify its functionality, and debug any issues.
The recipes can be executed at different time levels or can also be executed manually:

Setup occurs on a new instance after it successfully boots.

Configure occurs on all of the stack's instances when an instance enters or

leaves the online state.

Deploy occurs when you deploy an app.
Undeploy occurs when you delete an app.
Shutdown occurs when you stop an instance.
CloudWatch
Monitoring service provided by AWS which helps in gathering different data from our resources
and represent it in the form of graphs and enables us to set alarms for different kinds of
situations using SNS. Cloudwatch has a throughput to the limit to the data being sent which is
approximately 1 Mbps. Sometimes there may be a delay in the data arriving at Cloudwatch.
Logs in cloudwatch can be set for deletion after a particular interval. Filter types for logs
supported are Literal Terms, Common Log Format, JSON.
Data is gathered from different sources:
1. Default: By Default cloudwatch gathers data like CPU Utilization, Memory
Utilization, Network in/out, Read/Write to disks etc, these logs are collected by default
represented in the form of graphs and can be used in making relevant decisions in the
planning process, alarms can also be set on these based upon user defined matrices.
2. EC2 internal logs (User Defined): Various logs files generated inside of ec2
instances (ex: /var/log/secure) or custom logs for our application (ex:
/var/log/apache2/app.log) can also be transmitted to cloudwatch. An agent has to be

installed on the instance/s for this purpose and configured to send user defined logs to
cloudwatch. In the cloudwatch panel we can easily access these logs data as raw,
graphs or can also set alarms based upon our own custom parameters (ex: error, debug,
custom strings, HTTP responses etc.) to send us notifications based upon SNS topics
we define.
3. S3 logs (bucket logs): S3 buckets flag for logging can also be set and a cron job
can be set in an ec2 instance to read the logs from the s3 bucket and then send it to
cloudwatch (make sure the instance as sufficient permissions to do so), as the data is
sent to cloudwatch we can then access it using cli or by console as well as set alarms
and view GUI graphs for the same.
4. CloudTrail logging: AWS service CloudTrail now also comes with the the support
for cloudwatch by which we can provision CloudTrail can send logs to cloudwatch.
These logs collected are used by different services:
1. Used in generating graphical UI representation in the form of graphs.
2. Used in setting up alarms for notifications.
3. Used by AWS service known as Advisor for giving recommendation to improve
cost, performance and security of our cloud infrastructure.
4. Cloudwatch data can also be gathered using cli, api and stored in files like xlsx or
can be used by applications for processing.
5. Integration with other (third party) monitoring tools.
CloudTrail
While using different services offered by amazon like S3, Database, Cloudfront etc. we access
these resources by making API calls. CloudTrail provides us the feature to get the details of all
these activities in the form of log files which can be used for troubleshooting, compliance,
security, analysis and monitoring. These logs are stored in S3 bucket defined by the user. These
logs can also be sent to Cloudwatch for analysis and setting up notifications.
It answers the questions like:
Who made the API call, When made the API call, What was the API call, Which resources acted
upon in the API call, Where the API call made from and made to.
During Creation process:
1. Mention the name of the Cloudtrail and define whether to create a new bucket in
S3 or to use and existing bucket for storing logs. This process can also be done using
cli.

The logs stored by CloudTrail in S3 can also be configured to get moved to Glacier after a
certain time period for compliance.
CloudTrail can also be configured to send logs from multiple aws accounts to a single bucket.
Files will be arranged per account and region in S3 bucket for easier future access.
By Default CloudTrail encrypts logs stored at S3 using S3 server side encryption. We can also
define a user based key (KMS) that will be used for encryption and decryption.
CloudTrail also enables us to validate the log files stored in S3 are not changed. Detect whether
a log file was deleted or modified or changed. Cloudtrail will send digest file per hour for
validating the authenticity the digest file, digest file contains the hash value for the log files.
being sent will be signed by cloudtrail.
Cloudtrail also supports the integration of third party softwares for accessing these log files and
generating reports and performing analysis.
Simple Notification Service
This is the notification service provided by amazon by which the users subscribed can get
notification for our custom defined topics (events) through email, sms, etc. Can be used through
api to engage customers (ex: mobile apps using sns to mobile). It is a public subscribed
messaging service. The way it works is:
1. Create Topic: Create a new topic give any name to it.
2. Choose subscribers: Click subscribers and the format in which they will be
notified i.e. email, sms, http etc.
3. Once the user subscribes to the topic we can then add content to our sns topic
which will then get pushed to the subscriber by calling the topic through api or if an event
occurs which triggers the topic to send notification.
Types of publish:
1. Direct Publish(available to mobile devices only at this point): Pushes topic to a
wide variety of mobile devices whether it is iOS, Android, Kindle etc. without the concern
of using their respective push notification apis. (ex: create an application that pushes
notification to the user using sns)
2. Topic Publish: Publish topic to a wide variety of devices like sqs, email, http, sms
etc using filtering mechanisms provided by sns. (ex: users subscribe to sms notification
of a topic, now we can publish notification to those set of devices like mobiles)
SNS also accompanies well with S3 buckets and can be used to send push notifications based
upon different events for various S3 buckets like a notification system. SNS can also be

accompanied with cloudwatch which will notify user if a particular threshold exceeds and
accompanying SNS additionally with lambda function for an output functionality we wish. This
helps in additionally provisioning resources like ec2, storage etc. in case any event occurs.
Simple Email Service
Email service provided by amazon which can be used to send large number of emails using
email infrastructure provided by amazon with zero maintenance and troubleshooting. This can
be used to run email campaigns, newsletters etc. This service is used to send and receive
emails. This service can also be used by either Management console, cli or apis. The ses
service can be merged with your existing email id as the SMTP server (email sending server) by
which the emails send from your mail id will be labelled FROM address as your legitimate mail
id but the email will be sent from amazon ses servers.
The mail that gets bounced or are flagged by users as Spam reaches to their corresponding
ISPs. The ISPs then returns the notification back to SES which then returns the notification
back to us through email and can be configured to use SNS as well.
The email can also be configured to trigger an event in which case as any such email is
received it will execute our predefined set of codes which can be done using Lambda.
The Bounced or marked as Spam emails should be taken with high priority to make sure that
the reputation for your emails sent can be maintained and our emails reaching to customers
doesn't gets counted as Spam by Spam filters. Best practise is to stop sending email further to
such email ids.
To use SES:
1. Make sure to verify your sender email address (through verification email) or
domain name (done by adding TXT record to domain name provided by AWS)
2. Request for production access so that you can send email to large number of
users otherwise it will be in sandbox mode in which we can only send email to
authorised emails only.
3. Get the unique Url, Username and Password for SES and use these credentials
to send large number of emails either by using mail clients like Thunderbird or Outlook,
or these credentials can also be used in application code to send out emails through
code using ses apis.
Authenticate email sent by sender
1. DKIM: adds encrypted signature to your email which is used by ISP to
authenticate your email being sent. Proves to recipient that sender of email is legitimate,
also proves email content has not been tampered with, help ISP to associate reputation

with your domain. Setup using EasyDKIM and add TXT record to DNS records, enables
automatic key rotation.
2. SPF: Tells recipient which ip address are authorised to send email from your
domain name. Done by adding DNS TXT record consisting of authorised ip addresses.
Please note daily sending quota and maximum size for sending email through ses. Request for
additional limit increase if required.
Use mailbox simulator for load testing. Also test email sizes. Also test your feedback system for
increased load.
Use HTTP instead of SMTP to send email to SES to reduce number of connections required to
and fro to send an email also use persistent connection i.e. sending all the email through a
single connection instead of opening a new connection for every email being sent. Send in
parallel to send email to a large number of users using multiple processes/threads.
Simple Queue Service
Distributed queue system which allows applications to queue messages that one component in
the application generates to be consumed by another component. With SQS we can decouple
components of our applications and make them work independently. It helps in situation where
producer is creating more messages than being processed by the consumer. SQS also allows
multiple resources (producers/consumers) to read and write to the queue. No need for these
components (resources) to coordinate with each other.
General SQS operations:
1.
2.
3.
4.
5.
6.

Create Queue: Creates Queue for storing messages

Create Message: Creates message inside the queue for processing
Read Message: Reads message from the queue.
Delete Message: Deletes message from the queue.
Purge: Purges all the messages from the queue.
Delete Queue: Deletes the queue.

SQS also supports batch processing by which instead of retrieving one message at a time it
reads multiple messages at once from the queue. Visibility timeout is defined by the user as the
time between producer adding message to the queue, consumer reading the message and
processing it and deleting it from the queue between the visibility timeout period.
1. A system that needs to send a message will find an Amazon SQS queue, and
use SendMessage to add a new message to it.

2. A different system that processes messages needs more messages to process,

so it calls ReceiveMessage, and this message is returned.
3. Once a message has been returned by ReceiveMessage, it will not be returned
by any other ReceiveMessage until the visibility timeout has passed. This keeps multiple
computers from processing the same message at once.
4. If the system that processes messages successfully finishes working with this
message, it calls DeleteMessage, which removes the message from the queue so no
one else will ever process it. If this system fails to process the message, then it will be
read by another ReceiveMessage call as soon as the visibility timeout passes.
5. If you have associated a Dead Letter Queue with a source queue, messages will
be moved to the Dead Letter Queue after the maximum number of delivery attempts you
specify have been reached.
ElastiCache
AWS managed service that lets you easily create, use and resize cache clusters in the cloud.
The service improves the performance of web applications by allowing you to retrieve
information from fast, managed, in-memory caches, instead of relying entirely on slower diskbased databases. ElastiCache supports two open-source in-memory caching engines:
1. Memcached - a widely adopted memory object caching system. ElastiCache is
protocol compliant with Memcached, so popular tools that you use today with existing
Memcached environments will work seamlessly with the service. Patterns for sharding,
no persistence, multi threaded, supports strings, objects, slab allocator.
2. Redis a popular open-source in-memory key-value store that supports data
structures such as sorted sets and lists. ElastiCache supports Master / Slave replication
and Multi-AZ which can be used to achieve cross AZ redundancy. Single threaded, read
replicas, supports data types, more like Nosql db, pub/sub functionality, persistence,
atomic operations.
1. Select Engine (Memcached/ Redis)
2. Choose (version, port, parameter group, cluster name, node type) + redis options (multi AZ &
replication, S3 backup location)
3. Choose (subnet group, AZ, Security group, maintenance windows, SNS) + redis option
(Enable backups)
We obtain are: Endpoints
Endpoint: Refer to particular node in your cluster

Primary Endpoint: For redis replication group, a consistent DNS alias that refers to node where
writes should be sent
Configuration Endpoint: For memcached, a consistent DNS alias to query for the list of current
nodes participating in the cluster
Multi-AZ for Redis: Using this approach a redis node can be made primary and all other redis
nodes in different az as its read replicas, all the write queries goes to the primary node and
replicated to the read replicas, provides a highly available architecture.
Connecting to redis and memcached: either telnet or using client libraries different for different
programming languages, aws also provides its own libraries to connect to caching servers.
Sharding:
1. Modulo based upon number of servers: not a good approach as alot keys get
reassigned when we add a server to the cluster
2. Consistent hashing: a better approach. Consider a ring, divide it into slots,
servers are mapped to slots.
Memcache divide the memory in the form of slots with each slot having a variable amount of
chunk size, during the write operation the data is written to the slot which is the most compatible
by its chunk size, this does waste some memory but provides us with a caching ability, In case
no free memory is available to store more data than the least recently used (LRU) object is
removed from cache and new data is written.
Monitoring can be done using phpMemcache UI or by using CloudWatch Matrices
Caching can be architectured based upon different requirements and scenarios. Basic scenario
uses cache in congestion with application server in a simple manner.
Starting with lazy caching which is highly beneficial for read intensive patterns. Writes and
updates are performed in real time using database. Handles user update to their profile.
Elastic Transcoder
Transcoding service which enables us to transcode videos. It has three main components:
1. Input Bucket: This is the bucket from which Transcoder accesses the file.
2. Pipeline: This pipeline is used to transcode the supplied video file based upon
user defined preset.
3. Output Bucket: This bucket is used to store transcoded videos.

During the time of creation of the pipeline we define the pipeline name, input bucket and the
output bucket with the IAM role for the transcoder. After creating the pipeline we create a new
job and define the input video file from the input bucket, also mention the preset we want to
convert it to with the output file name and execute this job. After the job is complete we can get
the transcoded file from the output bucket.
We can also customize the pre existing presets (aspect ratio, frame rate etc.) and create our
own custom preset to be used by transcoder.
Transcoder also support parallel transcoding and multiple format transcoding of a single file at
the same time in parallel(4 at most from console). It also supports visual watermarks, thumbnail
generation, notification using sns, encryption. Elastic transcoder also supports creation of
playlist by which we can add different presets of transcoded file and view the playlist later, can
also be used for adaptive bitrate streaming of videos. It does not support live media transcoding
it only supports file based transcoding. To stream the media we can either use AWS Cloudfront
for HTTP distribution of video or setup server like IIS which can also be used for video playback.
Video playback directly from S3 is not supported.
On-Demand Streaming
In the on-demand streaming case, your video content is stored in S3. Your viewers can choose
to watch it at any desired time, hence the name on-demand. A complete on-demand streaming
solution typically makes use of Amazon S3 for storage, the Amazon Elastic Transcoder for video
processing, and Amazon CloudFront for delivery.
Once uploaded, you may need to convert your video into the size, resolution, or format needed
by a particular video player or mobile device. The Amazon Elastic Transcoder will take care of
this for you. The Elastic Transcoder takes content from one S3 bucket, transcodes it per your
request, and stores the result in a second S3 bucket. Again, you can manage this process from
the AWS Management Console, the command line, or via the Elastic Transcoder APIs.
With your content safely stored and available in the formats required by your users, the next
step is global delivery. This is a job for Amazon CloudFront. This service gives you scalable
global delivery at a low price point. Because CloudFront caches content at the edges, your
customers will experience uninterrupted video playback with minimal delays due to buffering.
At this point our storyline forks, and you have two options. You can deliver the entire video file
to the device before playing it, or you can stream it to the device.

The first option is very easy to implement and is supported by just about every mobile device
and desktop. All you need to do is to put your content in an S3 bucket and create a CloudFront
distribution that points to the bucket. Your users video player will use CloudFront URLs
(accessible as part of the distribution) to request the video file. The request will be directed to
the best edge location, based on the users location. CloudFront will serve the video from its
cache, fetching it from the S3 bucket if it is not already cached. This option has a couple of
downsides. It makes inefficient use of your viewers bandwidth. If the user doesnt bother to
watch the entire video, content would never be seen is still downloaded. Skipping ahead or fastforwarding also necessitates downloading of content that may never be seen.
The second option is almost always preferred. A newer family of video streaming protocols
including Apples HTTP Live Streaming (HLS), Microsofts Smooth Streaming, (SS) and
AdobesHTTP Dynamic Streaming (HDS) improve the user experience by delivering video as it
is being watched, generally fetching content a few seconds ahead of when it will be needed.
Playback starts more quickly, fast-forwarding is more efficient, and the overall user experience is
smoother. Mobile users appreciate this option because it doesnt waste their bandwidth and they
get to see the desired content more quickly.
You will need to do a little more work in order to implement the second option. Use the Elastic
Transcoder to convert your video files to HLS format (the most widely supported streaming
protocol). This will split the video into short segments, and will also create a manifest file. The
player uses the manifest file to fetch and play the segments as needed.

EC2 Container Service

Amazon EC2 Container Service (Amazon ECS) is a highly scalable, fast, container
management service that makes it easy to run, stop, and manage Docker containers on a
cluster of Amazon EC2 instances. Amazon ECS lets you launch and stop container-enabled
applications with simple API calls, allows you to get the state of your cluster from a centralized
service, and gives you access to many familiar Amazon EC2 features.
You can use Amazon ECS to schedule the placement of containers across your cluster based
on your resource needs, isolation policies, and availability requirements. Amazon ECS
eliminates the need for you to operate your own cluster management and configuration
management systems or worry about scaling your management infrastructure.
Amazon ECS contains the following components:

1. Cluster: A logical grouping of container instances that you can place tasks on.
2. Container instance: An Amazon EC2 instance that is running the Amazon ECS
agent and has been registered into a cluster.
3. Task definition: A description of an application that contains one or more
container definitions. A task definition is required to run Docker containers in Amazon
ECS.
4. Scheduler: The method used for placing tasks on container instances.
5. Service: An Amazon ECS service allows you to run and maintain a specified
number of instances of a task definition simultaneously.
6. Task: An instantiation of a task definition that is running on a container instance.
7. Container: A Linux container that was created as part of a task.
In summary, ECS enables us to run our application which is made up of different docker
containers each having its own set of properties and set of tasks that it can execute. Task
Definition defines these different containers and their properties which are used in our
application. We can execute as many tasks inside each container as we require. We can also
schedule the tasks to either run first time during execution or to run indefinitely in which case if
the task fails inside a container ECS will kill the failing task and re run the task again or will try to
re run until the threshold value. We can define the resources to be associated with each
container like cpu and memory during while defining the task definition. ECS has API for long
running applications and services In the create service option just reference the task definition
and the number of tasks you want to run.
Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between
different AWS compute and storage services, as well as on-premise data sources, at specified
intervals. With AWS Data Pipeline, you can regularly access your data where its stored,
transform and process it at scale, and efficiently transfer the results to AWS services such as
Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon Elastic MapReduce (EMR).
AWS Data Pipeline helps you easily create complex data processing workloads that are fault
tolerant, repeatable, and highly available. You dont have to worry about ensuring resource
availability, managing inter-task dependencies, retrying transient failures or timeouts in
individual tasks, or creating a failure notification system.
AWS Data Pipeline allows you to take advantage of a variety of features such as scheduling,
dependency tracking, and error handling. You can use activities and preconditions that AWS
provides and/or write your own custom ones. This means that you can configure an AWS Data

Pipeline to take actions like run Amazon EMR jobs, execute SQL queries directly against
databases, or execute custom applications running on Amazon EC2 or in your own datacenter.
AWS Data Pipeline handles:

Your jobs' scheduling, execution, and retry logic

Tracking the dependencies between your business logic, data sources,

and previous processing steps to ensure that your logic does not run until all of its
dependencies are met

Sending any necessary failure notifications

Creating and managing any temporary compute resources your jobs may
require
(ex: create data pipeline, which uses an EMR cluster, read apache web access logs, reformat
the log output to amazon S3)
By using Data Pipeline we can create a template (using GUI interface) or use an existing
template and can define the data flow between the resources. We can easily define the source
of the data (S3, DynamoDB etc.) and process that data (copy, transform etc. using EC2, EMR,
redshift and many more activities) and can load the transformed data into storage services like
S3, DynamoDB etc. We can easily schedule these data driven jobs using Data Pipeline(min 15
minutes interval). Supports SNS and logging of activities being carried out.

Elasticsearch Service
Amazon Elasticsearch Service is a managed service that makes it easy to deploy, operate, and
scale Elasticsearch in the AWS Cloud. Elasticsearch is a popular open-source search and
analytics engine for use cases such as log analytics, real-time application monitoring, and
clickstream analytics.
Elasticsearch can be easily combined with logstash for data ingestion and kibana for data
visualization. The elk (elasticsearch, logstash, kibana) stack provides tool for real time analytics
and data visualization.
Two types of nodes in elasticsearch cluster:
1. Data nodes: these nodes are responsible for handling the data and performing
any query on the data.
2. Master node: responsible for holding cluster stability and managing.
By enabling zone awareness feature the nodes will be distributed between multi az for high
availability.

The index is a collection of data which in turn comprises of many shards (instance of lucence
with a portion of an index) and with each shard having a number of documents associated with
unique id. Each document comprises of collection of fields.
The shards are distributed between the nodes of the cluster for analysis. Each shard also has a
backup of it running inside a different instance known as replica for in case if the primary shard
goes down the replica can be promoted as primary.
Data can be loaded either by using elasticsearch api or using logstash for log based or event
based data or lambda by inputting data from sources like DynamoDB, S3 and Kinesis and
loading it into lambda and then into elasticsearch for analysis. Cloudwatch can also be
configured to push its logs to elasticsearch for analysis and viewing.
ElasticSearch supports automated daily snapshots which are retained for a duration of 14 days.
Manual snapshots also supported that are stored in S3 and can be taken based upon our own
time duration.
Kinesis
For streaming data ingestion and processing.
The producers continually push data to Streams and the consumers process the data in real
time. Consumers can store their results using an AWS service such as Amazon DynamoDB,
Amazon Redshift, or Amazon S3. It has 4 use case scenarios.
The first scenario is around making the task of "Ingest Transform Load" a bit easier by handling
typical issues like scale, reliability, and durability of the underlying infrastructure. This is part of
being a managed service. Please bear in mind that as with all systems, our design choices can
lead to situations that can be construed to be limitations for other use cases. An example being:
a single data blob that is put into an Amazon Kinesis stream can be no larger than 50KB in size
in this initial release. So even though the aggregate stream can be arbitrarily large, what it
means is that if in your model, let's say the data record of interest is a few MB large - then it will
have to be chunked up into smaller pieces <=50KB before being put into the Amazon Kinesis
stream.
The second category is metrics extraction, and real-time KPI generation. Once data is ingested
continuously, there is always the desire to start running some useful application that can
generate some insights from the data. This data could then drive a dashboard or even be
emitted continuously into another store such as DynamoDB or Cassandra or anything else.

The third category is to run full on analytic applications that are operating on streams of data.
This could be a fraud application, network security/ intrusion detection algorithm, 'next-bestclick' evaluation, or others. Frequently this application will be part of a larger application stack.
The final category is to create more complex stream processing applications. An initial Amazon
Kinesis application can emit data - after say pre-processing and filtering the primary stream and put data into another stream that then drives other processing applications. In this was a
DAG of streams and processing stages could be created.
Simple Workflow Service
It is a task coordination and state management service for cloud applications. Allows us to
define a step by step workflow of the application. Each one of these steps can be broken as
tasks or components. Each components has workers that execute these instructions. It is a
distributed service which allows us to break our application on the basis of tasks to be carried
out and each of these tasks can be executed using simple workflow service. Each of these
components can be scaled up or down as per requirements.
Domain is used to determine scope of workflows
Multiple workflows can be present inside a single domain
Workflows cannot communicate with other workflows in different domain
Worker carries out the tasks assigned to it by the decider and returns the result back to the
decider
Activity Worker - Worker that performs an activity/task that is part of the workflow.
Activity worker poll SWF for new tasks it has to perform
After receiving the task the worker will execute the task and report back to swf
workers can consist of EC2 to carry out the task or can also use human to perform the task
Activity Task - Task assigned to worker that tells what activity to perform like encode video or
check inventory
Decision Task - Tells the decider that state of workflow execution has changed. Allows decider to
determine what next activity has to be performed.
Config
AWS Config is a fully managed service that provides you with an AWS resource inventory,
configuration history, and configuration change notifications to enable security and governance.
With AWS Config you can discover existing AWS resources, export a complete inventory of your

AWS resources with all configuration details, and determine how a resource was configured at
any point in time. These capabilities enable compliance auditing, security analysis, resource
change tracking, and troubleshooting.
DirectConnect
Service that enables to connect to our cloud infrastructure from existing physical infrastructure
using a direct link. This reduces removes the traffic going to internet and provides security and
stable high speed connectivity.
A single direct connection can be configured with multiple virtual interfaces with each connecting
to different VPCs in our AWS infrastructure.