AWS CodePipeline

AWS CodePipeline

Overview

The following document is a step-by-step guide on how to create a CI/CD pipeline for an EC2 instance.

diagram

Components

Here are some of the main services/stages for the pipeline.

CodeCommit (source)

This a source control repository, exactly similar to GitHub. It'll store your project as a repo and trigger the pipeline whenever new changes are pushed to deploy our code automatically.

We're not limited to CodeCommit to store our project, we connect an external GitHub repo with the pipeline,

CodeArtifact

A service to store, manage, and publish software packages used in our application during development process.

Think of it as a centralized repository where you can store and manage your software packages, such as libraries, frameworks, and other dependencies that your application relies on. Instead of manually downloading and installing these packages on each developer's machine or on different servers, you can use CodeArtifact to store and manage them in one place.

CodeBuild (build)

Takes care of building, testing, and compiling our application. This stage or service happen before the deploying stage.

CodeDeploy (deploy)

A service used to deploy our application to EC2 instances, Lambda functions, and on-premises instances.

on-premises instances: any physical device that is not an Amazon EC2 instance that can run the CodeDeploy agent and connect to public AWS service endpoints

You can specify the deployment configuration, deployment group, and revision of your application code using YAML file. CodeDeploy will then handle the deployment process, including copying your code to the target environment, running pre and post deployment tests, and rolling back if there are any issues.

CodePipeline

A service that provides continuous delivery and release automation capabilities for your application code. It allows you to build, test, and deploy your code changes automatically

CodePipeline will then handle the pipeline execution, including building and testing your code using AWS CodeBuild, deploying your code using AWS CodeDeploy, and automating the release process.

CDK code

Steps

  1. Create an EC2 instance
    • Install codedeploy-agent by a user data script. Without codedeploy-agent, CodeDeploy service can't deploy to our EC2 instance

const createEC2Instance = (parentThis: Construct) => {
  // create VPC in which we'll launch the Instance
  const vpc = new ec2.Vpc(parentThis, 'my-cdk-vpc', {
    ipAddresses: ec2.IpAddresses.cidr(VPC_IP),
    natGateways: 0,
    subnetConfiguration: [
      { name: 'public', cidrMask: 24, subnetType: ec2.SubnetType.PUBLIC },
    ],
  })


  // create a key pair for ssh access
  const keyPair = new ec2.CfnKeyPair(parentThis, 'key-pair', {
    keyName: 'MyKeyPairB',
  })


  const role = new iam.Role(parentThis, 'MyRole', {
    assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
    managedPolicies: [
      iam.ManagedPolicy.fromManagedPolicyArn(
        parentThis,
        'ManagedPolicy',
        'arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole'
      ),
      iam.ManagedPolicy.fromManagedPolicyArn(
        parentThis,
        'ManagedPolicy2',
        'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforAWSCodeDeploy'
      ),
    ],
  })


  // create the EC2 Instance
  const ec2Instance = new ec2.Instance(parentThis, 'ec2-instance', {
    vpc,
    vpcSubnets: {
      subnetType: ec2.SubnetType.PUBLIC,
    },
    instanceType: ec2.InstanceType.of(
      ec2.InstanceClass.T2,
      ec2.InstanceSize.MICRO
    ),
    machineImage: new ec2.GenericLinuxImage({
      // ubuntu image AMI ID
      // from https://cloud-images.ubuntu.com/locator/ec2/
      'us-east-1': 'ami-0557a15b87f6559cf',
    }),
    keyName: keyPair.keyName,
    role,
    instanceName: 'CdkInstanceVfinal',
  })


  // bash script to quickly install the codedeploy-agent
  // what is used by CodeDeploy to deploy our code to the EC2 instance
  const userDataScript = readFileSync('./lib/startup.sh', 'utf8')
  // add the User Data script to the Instance
  ec2Instance.addUserData(userDataScript)


  return { ec2Instance, vpc }
}

  1. Create a target group

const createTargetGroup = (
  parentThis: Construct,
  vpc: cdk.aws_ec2.IVpc,
  ec2Instance: cdk.aws_ec2.Instance
) => {
  return new NetworkTargetGroup(parentThis, 'target-group', {
    // Has to be TCP in order to work with the load balancer
    protocol: Protocol.TCP,
    port: 80,
    targetType: TargetType.INSTANCE,
    healthCheck: {
      protocol: Protocol.HTTP,
      path: '/health',
    },
    // EC2 instances
    targets: [new targets.InstanceTarget(ec2Instance)],
    vpc,
  })
}

  1. Create a load balancer

const createNlb = (
  parentThis: Construct,
  vpc: cdk.aws_ec2.Vpc,
  targetGroup: cdk.aws_elasticloadbalancingv2.NetworkTargetGroup
) => {
  // create a network load balancer
  const nlb = new NetworkLoadBalancer(parentThis, 'nlb', {
    loadBalancerName: 'MyLoadBalancer-cdk',
    vpc,
    internetFacing: true,
  })


  // create a new listener
  const listener = nlb.addListener('Listener', {
    port: 80,
    protocol: Protocol.TCP,
    defaultTargetGroups: [targetGroup],
  })


  return { nlb, listener }
}

  1. Create a remote repository in CodeCommit

const createCommitRepo = (parentThis: Construct) =>
  new codecommit.Repository(parentThis, 'CodeCommitRepo', {
    repositoryName: 'MyCdkRepo',
  })

  1. Create a deployment application in CodeDeploy
    • An application tells CodeDeploy what to deploy and how to deploy it to EC2/On-Premises, AWS Lambda, and ECS
    • Configure it to deploy to our EC2 instance and use the same load balancer as the EC2 instance

const application = new codedeploy.ServerApplication(
  parentThis,
  'CodeDeployApplication'
)

  1. Create a deployment group in CodeDeploy
    • Deployment group list an application's deployment groups which include details about a target environment, how traffic shifts during a deployment, and monitoring settings

const deploymentGroup = new codedeploy.ServerDeploymentGroup(
  parentThis,
  'CodeDeployDeploymentGroup',
  {
    application,
    deploymentGroupName: 'MyDeploymentGroup',
    // adds EC2 instances matching tags
    ec2InstanceTags: new codedeploy.InstanceTagSet({
      // any instance with tags satisfying
      // will match this group
      Name: ['CdkInstanceVfinal'],
    }),
    loadBalancer: codedeploy.LoadBalancer.network(targetGroup),
    // auto creates a role for it
    // role: <someRole>
  }
)

  1. Create a pipeline in CodePipeline
    • Tell the pipeline to get our project code from CodeCommit repo in the source stage
    • Tell the pipeline to deploy our code through CodeDeploy

const setupCodePipeline = (
  parentThis: Construct,
  application: cdk.aws_codedeploy.ServerApplication,
  deploymentGroup: cdk.aws_codedeploy.ServerDeploymentGroup,
  commitRepo: cdk.aws_codecommit.Repository
) => {
  const pipeline = new codepipeline.Pipeline(parentThis, 'MyPipeline', {
    // If this is true, our CodeCommit repo in S3 will be encrypted
    // and the deploying stage will fail because it can't decrypt the files
    // couldn't find a solution yet
    crossAccountKeys: false,
  })


  // create source & deploy stages
  const sourceStage = pipeline.addStage({
    stageName: 'Source',
  })
  const deployStage = pipeline.addStage({
    stageName: 'Deploy',
  })


  const sourceOutput = new codepipeline.Artifact('source')


  const sourceStageAction = new codepipelineActions.CodeCommitSourceAction({
    repository: codecommit.Repository.fromRepositoryName(
      parentThis,
      'CodeRepo',
      commitRepo.repositoryName
    ),
    actionName: 'Source',
    // based on my understanding, it stores the project code
    // after pulling/ cloning it into a store(artifact)
    // then the deploy stage takes the artifact or our project code as an input
    output: sourceOutput,
  })


  // set the action of what to happen in this stage
  sourceStage.addAction(sourceStageAction)


  const deployStageAction =
    new codepipelineActions.CodeDeployServerDeployAction({
      deploymentGroup:
        codedeploy.ServerDeploymentGroup.fromServerDeploymentGroupAttributes(
          parentThis,
          'Deployment-group',
          {
            application,
            deploymentGroupName: deploymentGroup.deploymentGroupName,
          }
        ),
      actionName: 'Deploy',
      input: sourceOutput,
    })


  // set the action of what to happen in this stage
  deployStage.addAction(deployStageAction)


  return pipeline
}

Code

import * as cdk from 'aws-cdk-lib'
import * as ec2 from 'aws-cdk-lib/aws-ec2'
import * as targets from 'aws-cdk-lib/aws-elasticloadbalancingv2-targets'
import {
  NetworkLoadBalancer,
  NetworkTargetGroup,
  Protocol,
  TargetType,
} from 'aws-cdk-lib/aws-elasticloadbalancingv2'
import * as codedeploy from 'aws-cdk-lib/aws-codedeploy'
import * as codecommit from 'aws-cdk-lib/aws-codecommit'
import * as codepipeline from 'aws-cdk-lib/aws-codepipeline'
import * as codepipelineActions from 'aws-cdk-lib/aws-codepipeline-actions'
import * as iam from 'aws-cdk-lib/aws-iam'
import { Construct } from 'constructs'
import { readFileSync } from 'fs'


const VPC_IP = '10.0.0.0/16'


const createEC2Instance = (parentThis: Construct) => {
  // create VPC in which we'll launch the Instance
  const vpc = new ec2.Vpc(parentThis, 'my-cdk-vpc', {
    ipAddresses: ec2.IpAddresses.cidr(VPC_IP),
    natGateways: 0,
    subnetConfiguration: [
      { name: 'public', cidrMask: 24, subnetType: ec2.SubnetType.PUBLIC },
    ],
  })


  // create a key pair for ssh access
  const keyPair = new ec2.CfnKeyPair(parentThis, 'key-pair', {
    keyName: 'MyKeyPairB',
  })


  const role = new iam.Role(parentThis, 'MyRole', {
    assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
    managedPolicies: [
      iam.ManagedPolicy.fromManagedPolicyArn(
        parentThis,
        'ManagedPolicy',
        'arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole'
      ),
      iam.ManagedPolicy.fromManagedPolicyArn(
        parentThis,
        'ManagedPolicy2',
        'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforAWSCodeDeploy'
      ),
    ],
  })


  // create the EC2 Instance
  const ec2Instance = new ec2.Instance(parentThis, 'ec2-instance', {
    vpc,
    vpcSubnets: {
      subnetType: ec2.SubnetType.PUBLIC,
    },
    instanceType: ec2.InstanceType.of(
      ec2.InstanceClass.T2,
      ec2.InstanceSize.MICRO
    ),
    machineImage: new ec2.GenericLinuxImage({
      // ubuntu image AMI ID
      // from https://cloud-images.ubuntu.com/locator/ec2/
      'us-east-1': 'ami-0557a15b87f6559cf',
    }),
    keyName: keyPair.keyName,
    role,
    instanceName: 'CdkInstanceVfinal',
  })


  // bash script to quickly setup the server
  const userDataScript = readFileSync('./lib/startup.sh', 'utf8')
  // add the User Data script to the Instance
  ec2Instance.addUserData(userDataScript)


  return { ec2Instance, vpc }
}


const createTargetGroup = (
  parentThis: Construct,
  vpc: cdk.aws_ec2.IVpc,
  ec2Instance: cdk.aws_ec2.Instance
) => {
  return new NetworkTargetGroup(parentThis, 'target-group', {
    // Has to be TCP in order to work with the load balancer
    protocol: Protocol.TCP,
    port: 80,
    targetType: TargetType.INSTANCE,
    healthCheck: {
      protocol: Protocol.HTTP,
      path: '/health',
    },
    // EC2 instances
    targets: [new targets.InstanceTarget(ec2Instance)],
    vpc,
  })
}


const createNlb = (
  parentThis: Construct,
  vpc: cdk.aws_ec2.Vpc,
  targetGroup: cdk.aws_elasticloadbalancingv2.NetworkTargetGroup
) => {
  // create a network load balancer
  const nlb = new NetworkLoadBalancer(parentThis, 'nlb', {
    loadBalancerName: 'MyLoadBalancer-cdk',
    vpc,
    internetFacing: true,
  })


  // create a new listener
  const listener = nlb.addListener('Listener', {
    port: 80,
    protocol: Protocol.TCP,
    defaultTargetGroups: [targetGroup],
  })


  return { nlb, listener }
}


const setEc2SecurityGroup = (
  parentThis: Construct,
  vpc: cdk.aws_ec2.Vpc,
  ec2Instance: cdk.aws_ec2.Instance
) => {
  // create a Security Group for the Instance
  const SecurityGroup = new ec2.SecurityGroup(parentThis, 'ec2-sg', {
    vpc,
    allowAllOutbound: true,
  })


  // enable SSH
  SecurityGroup.addIngressRule(
    ec2.Peer.anyIpv4(),
    ec2.Port.tcp(22),
    'allow SSH access from anywhere'
  )


  // enable HTTP for load balancer
  SecurityGroup.addIngressRule(
    ec2.Peer.ipv4(VPC_IP),
    ec2.Port.tcp(80),
    'allow HTTP traffic from anywhere'
  )


  ec2Instance.addSecurityGroup(SecurityGroup)


  return SecurityGroup
}


const createCommitRepo = (parentThis: Construct) =>
  new codecommit.Repository(parentThis, 'CodeCommitRepo', {
    repositoryName: 'MyCdkRepo',
  })


const setupCodeDeployApp = (
  parentThis: Construct,
  targetGroup: cdk.aws_elasticloadbalancingv2.NetworkTargetGroup
) => {
  const application = new codedeploy.ServerApplication(
    parentThis,
    'CodeDeployApplication'
  )


  // create deployment group
  const deploymentGroup = new codedeploy.ServerDeploymentGroup(
    parentThis,
    'CodeDeployDeploymentGroup',
    {
      application,
      deploymentGroupName: 'MyDeploymentGroup',
      // adds EC2 instances matching tags
      ec2InstanceTags: new codedeploy.InstanceTagSet({
        // any instance with tags satisfying
        // will match this group
        Name: ['CdkInstanceVfinal'],
      }),
      loadBalancer: codedeploy.LoadBalancer.network(targetGroup),
      // auto creates a role for it
      // role: <someRole>
    }
  )


  return { application, deploymentGroup }
}


const setupCodePipeline = (
  parentThis: Construct,
  application: cdk.aws_codedeploy.ServerApplication,
  deploymentGroup: cdk.aws_codedeploy.ServerDeploymentGroup,
  commitRepo: cdk.aws_codecommit.Repository
) => {
  const pipeline = new codepipeline.Pipeline(parentThis, 'MyPipeline', {
    // If this is true, our CodeCommit repo in S3 will be encrypted
    // and the deploying stage will fail because it can't decrypt the files
    // couldn't find a solution yet
    crossAccountKeys: false,
  })


  // create source & deploy stages
  const sourceStage = pipeline.addStage({
    stageName: 'Source',
  })
  const deployStage = pipeline.addStage({
    stageName: 'Deploy',
  })


  const sourceOutput = new codepipeline.Artifact('source')


  const sourceStageAction = new codepipelineActions.CodeCommitSourceAction({
    repository: codecommit.Repository.fromRepositoryName(
      parentThis,
      'CodeRepo',
      commitRepo.repositoryName
    ),
    actionName: 'Source',
    // based on my understanding, it stores the project code
    // after pulling/ cloning it into a store(artifact)
    // then the deploy stage takes the artifact or our project code as an input
    output: sourceOutput,
  })


  // set the action of what to happen in this stage
  sourceStage.addAction(sourceStageAction)


  const deployStageAction =
    new codepipelineActions.CodeDeployServerDeployAction({
      deploymentGroup:
        codedeploy.ServerDeploymentGroup.fromServerDeploymentGroupAttributes(
          parentThis,
          'Deployment-group',
          {
            application,
            deploymentGroupName: deploymentGroup.deploymentGroupName,
          }
        ),
      actionName: 'Deploy',
      input: sourceOutput,
    })


  // set the action of what to happen in this stage
  deployStage.addAction(deployStageAction)


  return pipeline
}


export class CdkStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props)


    const { ec2Instance, vpc } = createEC2Instance(this)
    const targetGroup = createTargetGroup(this, vpc, ec2Instance)
    createNlb(this, vpc, targetGroup)


    setEc2SecurityGroup(this, vpc, ec2Instance)


    // CodeCommit
    const commitRepo = createCommitRepo(this)


    // CodeDeploy
    const { application, deploymentGroup } = setupCodeDeployApp(
      this,
      targetGroup
    )


    // CodePipeline
    setupCodePipeline(this, application, deploymentGroup, commitRepo)


    // outputs the repo URL in the terminal
    new cdk.CfnOutput(this, 'apiUrl', {
      value: commitRepo.repositoryCloneUrlHttp,
    })
  }
}

Demo

  1. Go to your poject folder and push to the repository clone URL, otherwise it'll be empty and you won't be able to deploy
    • git push https://git-codecommit.us-east-1.amazonaws.com/v1/repos/SuperRepo -all
    • You can clone my demo express.js project and then push it to the CodeCommit repo
  2. Go to the CodePipeline console and notice how it shows a triggered pipeline
  3. Once the pipeline finish executing successfully, go the EC2 instance URL and test it out

Potential issues you might encounter

  • CodeDeploy: if events are all pending, it means the codedeploy-agent wasn't installed in the EC2 instance
  • CodeDeploy: If AllowTraffic event fails, it means the EC2 instance isn't healthy, verify the target group and load balancer are configured correctly
  • CodePipeline: if the source stage fails, confirm you the codecommit repo isn't empty
  • CodeDeploy(EC2): if codedeploy-agent isn't installed, install it with a user data script (ec2)
  • EC2: if the new changes aren't deployed or seem like that, this probably means the node server process wasn't killed and therefore when trying to start a new server, it'll throw an error saying that port 80 is already used. So, make sure to kill all node processes first.
  • CodeDeploy: if you're stuck in AllowTraffic or BlockTraffic, this is normal because these take a few minutes
  • CodeDeploy: if you're stuck in AllowTraffic, this means there's something wrong with the target group. It could be that the target group isn't connected to a load balancer or your EC2 instance is an healthy

Important CodeDeploy events

  • BlockTraffic: in this event, CodeDeploy unregister the EC2 instance from the target group so it doesn't receive any requests Block
  • AllowTraffic: in this event, CodeDeploy register the EC2 instance and waits until it's considered healthy

References