Welcome to the third part of this serie! In this segment, we dive into testing and pipeline configuration on Google Cloud, specifically focusing on continuous integration using Cloud Build, On-demand Vulnerability Scanner, and Artifact Registry. You can find the project repository here, or, if you prefer, you can bring your own project.

Let me walk you through the pipeline. With each push to the main branch, Cloud Build is triggered. First, it runs unit tests on the code. If the tests pass, it proceeds to build the image. After the image is built, Cloud Build invokes the image scanner to ensure it's free of vulnerabilities. If all is well, the image is sent and stored in the Artifact Registry, ready for deployment. But for this article, we'll focus solely on the CI part. Let's start with the tests.

Unittest

Here's the code we plan to test

import os
from flask import Flask
from pymongo import MongoClient
from flask import Flask, render_template, request, url_for, redirect
from bson.objectid import ObjectId
import mongomock



app = Flask(__name__, template_folder='templates')

if os.environ.get('TESTING'):
    client = mongomock.MongoClient()
else:
    client = MongoClient(os.environ['MONGO_URI'])


db = client.flask_db
todos = db.todos


@app.route('/', methods=('GET', 'POST'))
def index():
    if request.method=='POST':
        content = request.form['content']
        degree = request.form['degree']
        todos.insert_one({'content': content, 'degree': degree})
        return redirect(url_for('index'))

    all_todos = todos.find()
    return render_template('index.html', todos=all_todos)


@app.post('/<id>/delete/')
def delete(id):
    todos.delete_one({"_id": ObjectId(id)})
    return redirect(url_for('index'))

For an explanation of the code, refer to the first article in this series.

Now, let's move on to the testing phase

The test is written using Python's built-in unittest module, which provides a framework for writing and running tests.

1. Import necessary modules and create a mock MongoDB instance

   The test begins by importing the necessary modules. unittest is the testing framework, patch and MagicMock from unittest.mock are used to replace parts of the system that you're testing with mock objects, and ObjectId from bson.objectid is used to create unique identifiers. The app and todos are imported from the app.py file. mongomock is used to create a mock MongoDB instance for testing, and flask is used to manipulate the request context during testing.

    import unittest
    from unittest.mock import patch, MagicMock
    from bson.objectid import ObjectId
    from app import app, todos
    import mongomock
    import flask
   
    mock_db = mongomock.MongoClient().db
   

2. Define the test case

   A test case is defined by creating a new class that inherits from unittest.TestCase. This class will contain methods that represent individual tests.

    class TestApp(unittest.TestCase):
   

3. Set up the test environment

   The setUp method is a special method that is run before each test. Here, it's used to create a test client instance of the Flask app and enable testing mode.

    def setUp(self):
        self.app = app.test_client()
        self.app.testing = True
   

4. Write the test

   The test_index_post method is the actual test. It tests the behavior of the app when a POST request is sent to the index route (/).

    def test_index_post(self):
   

5. Mock the database operation

   The patch function is used to replace the insert_one method of todos with a MagicMock. This allows the test to simulate the behavior of the database operation without actually interacting with a real database.

    with patch('app.todos.insert_one', new_callable=MagicMock) as mock_insert_one:
   

6. Create a test request context

   A test request context is created for the app using app.test_request_context. This allows the test to simulate a request to the app.

    with app.test_request_context('/'):
   

7. Set the request method and form data

   The request method is set to 'POST' and the request form data is set to a dictionary with 'content' and 'degree' keys.

    flask.request.method = 'POST'
    flask.request.form = {'content': 'Test Content', 'degree': 'Test Degree'}
   

8. Send a POST request to the app

   A POST request is sent to the app using self.app.post. The form data is passed as the data argument.

    result = self.app.post('/', data=flask.request.form)
   

9. Assert the expected results

   The assertEqual method is used to check that the status code of the response is 302. The assert_called method is used to check that the insert_one method was called.

    self.assertEqual(result.status_code, 302)
    mock_insert_one.assert_called()
   

This test ensures that when a POST request is sent to the index route with the correct form data, the app responds with a 302 status code and inserts the data into the database.

Your test code should look something like the following:

import unittest
from unittest.mock import patch, MagicMock
from bson.objectid import ObjectId
from app import app, todos
import mongomock
import flask

# Create a mock MongoDB instance
mock_db = mongomock.MongoClient().db

class TestApp(unittest.TestCase):
    def setUp(self):
        # Create a test client instance
        self.app = app.test_client()
        # Enable testing mode. Exceptions are propagated rather than handled by the the app's error handlers
        self.app.testing = True 

    def test_index_post(self):
        # Patch the insert_one method of todos with a MagicMock
        with patch('app.todos.insert_one', new_callable=MagicMock) as mock_insert_one:
            # Create a test request context for the app
            with app.test_request_context('/'):
                # Set the request method to 'POST'
                flask.request.method = 'POST'
                # Set the request form data
                flask.request.form = {'content': 'Test Content', 'degree': 'Test Degree'}
                # Send a POST request to the app
                result = self.app.post('/', data=flask.request.form)
                # Assert that the status code of the response is 302
                self.assertEqual(result.status_code, 302)
                # Assert that the insert_one method was called
                mock_insert_one.assert_called()

Now, to execute the test, set the environment variable TESTING=true, Setting TESTING=True switches the application to use a mock MongoDB client for testing, instead of the real MongoDB database.

Now, if your test is successful, let's move on to configuring Cloud Build.

Cloud Build setup

Follow the guide to connect Cloud Build to your repository and this one for initial configurations.

Once that's done, let's move on to writing the Cloud Build configuration file, where we'll instruct it on how to execute the pipeline, the steps involved, dependencies, and so on.

Cloud Build config file

The Cloud Build Config file is written in YAML, a human-readable data serialization language.

Here are the main sections of our config file:

1. Substitutions: These are user-defined variables that can be replaced in the Cloud Build configuration file. They are defined under the substitutions key. In this case, _REGION, _REPOSITORY, _IMAGE, and _SEVERITY are defined.

    substitutions:
      _REGION: us-central1
      _REPOSITORY: from-legacy-to-cloud
      _IMAGE: from-legacy-to-cloud
      _SEVERITY: '"CRITICAL|HIGH"'
   

2. Steps: These are the operations that Cloud Build will perform. Each step is a separate action and they are executed in the order they are defined.

   • Step 0: Install test dependencies: This step uses a Python 3.10 Docker image to install the test dependencies listed in docker/requirements-test.txt. The entrypoint is set to /bin/bash, which means that the command that follows will be executed in a bash shell. The args key specifies the command to be executed, which in this case is a pip install command. The -c flag tells bash to read commands from the following string. The | character allows us to write multiple commands, which will be executed in order.

       - name: 'python:3.10-slim'
         entrypoint: '/bin/bash'
         args:
           - '-c'
           - |
             pip install --user -r docker/requirements-test.txt
         id: 'install-test-dependencies'
     

   • Step 1: Run unit tests: This step also uses a Python 3.10 Docker image to run the unit tests defined in test.py. The export TESTING=True command sets an environment variable TESTING to True, which can be used to change the behavior of the application during testing. The cd docker command changes the current directory to docker, where the test file is located. The python -m unittest test.py command runs the unit tests in test.py.

       - name: 'python:3.10-slim'
         entrypoint: '/bin/bash'
         args:
           - '-c'
           - |
             export TESTING=True
             cd docker 
             python -m unittest test.py
         id: 'run-tests'
     

   • Step 2: Build the Docker image: This step uses the docker Cloud Builder to build a Docker image from the Dockerfile located in the docker/ directory. The image is tagged with the commit SHA. The waitFor key is used to specify that this step should wait for the run-tests step to complete before it starts. The args key specifies the command to be executed, which in this case is a docker build command. The -t flag is used to name and optionally tag the image in the 'name:tag' format.

       - name: 'gcr.io/cloud-builders/docker'
         args: ['build', '-t', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA', 'docker/']
         waitFor: ['run-tests']
         id: 'build-image'
     

• Step 3: Inspect the Docker image and write the digest to a file: This step uses the docker Cloud Builder to inspect the Docker image and write the image digest to a file. The image digest is a unique identifier for the image. The docker image inspect command retrieves detailed information about the Docker image. The --format option is used to format the output using Go templates. The {{index .RepoTags 0}}@{{.Id}} template retrieves the first tag of the image and the image ID. The > operator redirects the output to a file. The && operator is used to execute the cat command only if the previous command succeeded.

    - name: 'gcr.io/cloud-builders/docker'
      entrypoint: '/bin/bash'
      args:
        - '-c'
        - |
          docker image inspect $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA --format '{{index .RepoTags 0}}@{{.Id}}' > /workspace/image-digest.txt &&
          cat /workspace/image-digest.txt
      id: 'inspect-image'
  

• Step 4: Scan the Docker image for vulnerabilities: This step uses the cloud-sdk Cloud Builder to scan the Docker image for vulnerabilities. The scan ID is written to a file. The gcloud artifacts docker images scan command scans the Docker image for vulnerabilities. The --format='value(response.scan)' option is used to retrieve the scan ID from the response. The > operator redirects the output to a file.

    - id: scan
      name: gcr.io/google.com/cloudsdktool/cloud-sdk
      entrypoint: /bin/bash
      args:
      - -c
      - |
        gcloud artifacts docker images scan $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA \
        --format='value(response.scan)' > /workspace/scan_id.txt
  

• Step 5: Check the severity of any vulnerabilities found: This step uses the cloud-sdk Cloud Builder to list the vulnerabilities found in the Docker image and check their severity. If any vulnerabilities with a severity matching _SEVERITY are found, the build fails. The gcloud artifacts docker images list-vulnerabilities command lists the vulnerabilities found in the Docker image. The --format='value(vulnerability.effectiveSeverity)' option is used to retrieve the severity of each vulnerability. The grep -Exq $_SEVERITY command checks if any of the severities match _SEVERITY. The echo command prints a message and the exit 1 command terminates the build if a match is found.

    - id: severity check
      name: gcr.io/google.com/cloudsdktool/cloud-sdk
      entrypoint: /bin/bash
      args:
      - -c
      - |
        gcloud artifacts docker images list-vulnerabilities $(cat /workspace/scan_id.txt) \
        --format='value(vulnerability.effectiveSeverity)' | if grep -Exq $_SEVERITY; \
        then echo 'Failed vulnerability check' && exit 1; else exit 0; fi
  

• Step 6: Push the Docker image to Google Cloud Artifact Registry: This step uses the docker Cloud Builder to push the Docker image to the Google Cloud Artifact Registry. The waitFor key is used to specify that this step should wait for the severity check step to complete before it starts. The docker push command pushes the Docker image to a repository.

    - name: 'gcr.io/cloud-builders/docker'
      args: ['push', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA']
      id: 'push-image'
      waitFor: ['severity check']
  

• Images: This key specifies the Docker images that Cloud Build should build and push to the Google Cloud Artifact Registry. In this case, it's the Docker image built in Step 2.

    images:
    - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA'
  

This cloudbuild.yaml file defines a complete CI/CD pipeline for our application. It installs test dependencies, runs unit tests, builds a Docker image, inspects the image, scans the image for vulnerabilities, checks the severity of any vulnerabilities found, and pushes the image to the Google Cloud Artifact Registry. This pipeline ensures that the application is tested, secure, and ready for deployment.

The complete config file should look like this:

substitutions:
  _REGION: us-central1
  _REPOSITORY: from-legacy-to-cloud
  _IMAGE: from-legacy-to-cloud
  _SEVERITY: '"CRITICAL|HIGH"'

steps:
# Step 0: Install test dependencies
- id: 'install-test-dependencies'
  name: 'python:3.10-slim'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      pip install --user -r docker/requirements-test.txt

# Step 1: Run unit tests
- id: 'run-tests'
  name: 'python:3.10-slim'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      export TESTING=True
      cd docker 
      python -m unittest test.py

# Step 2: Build the Docker image
- id: 'build-image'
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA', 'docker/']
  waitFor: ['run-tests']

# Step 3: Inspect the Docker image and write the digest to a file.
- id: 'inspect-image'
  name: 'gcr.io/cloud-builders/docker'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      docker image inspect $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA --format '{{index .RepoTags 0}}@{{.Id}}' > /workspace/image-digest.txt &&
      cat /workspace/image-digest.txt

# Step 4: Scan the Docker image for vulnerabilities
- id: scan
  name: gcr.io/google.com/cloudsdktool/cloud-sdk
  entrypoint: /bin/bash
  args:
  - -c
  - |
    gcloud artifacts docker images scan $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA \
    --format='value(response.scan)' > /workspace/scan_id.txt

# Step 5: Check the severity of any vulnerabilities found
- id: severity check
  name: gcr.io/google.com/cloudsdktool/cloud-sdk
  entrypoint: /bin/bash
  args:
  - -c
  - |
    gcloud artifacts docker images list-vulnerabilities $(cat /workspace/scan_id.txt) \
    --format='value(vulnerability.effectiveSeverity)' | if grep -Exq $_SEVERITY; \
    then echo 'Failed vulnerability check' && exit 1; else exit 0; fi

# Step 6: Push the Docker image to Google Cloud Artifact Registry
- id: 'push-image'
  name: 'gcr.io/cloud-builders/docker'
  args: ['push', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA']
  waitFor: ['severity check']

images:
- '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA'

View build results

Now, commit and push your changes. If the Cloud Build triggers are configured correctly, the build should be triggered. Connect to the Google Cloud Console, go to Cloud Build > History to view your builds.

If it fails, click on it to see the error messages and troubleshoot to resolve the issues. Once the build succeeds, you can access the Artifact Registry and see the stored image, ready for use.

What next ?

Well, that wraps up this article. In the next one, we'll delve into automating deployments—the CD part. After vulnerability scanning of container images, we'll be putting security policies in place through Binary Authorization, allowing only approved/trusted images to be deployed on Cloud Run. But before that, we'll migrate our Mongo database to Google Firestore. After that, we'll deploy our app on Cloud Run and connect it to Firestore to make it fully operational.

See you in the next article. Until then, I'm available on social media (I'm more active on LinkedIn) for any information or additional suggestions. Thanks for reading!

ArgoCD is a Kubernetes-native CD tool that can help you to automate the deployment of your applications to Kubernetes. It is a declarative tool, which means that you can define the desired state of your applications in a Git repository. ArgoCD will then automatically synchronize the actual state of your applications with the desired state.

ArgoCD is a powerful tool that can help you to improve your CD process. It is easy to use, and it can be integrated with a wide range of other tools. If you are looking for a way to automate the deployment of your applications to Kubernetes, then ArgoCD is a great option.

In this blog post, we will explore the process of setting up continuous integration (CI) using GitHub Actions, and then we will delve into configuring ArgoCD to handle the continuous deployment (CD) aspect.

Why argoCD ?

For a brief overview of the benefits and reasons for using ArgoCD, I recommend checking out my LinkedIn post on the subject. In the post, I discuss the key advantages of leveraging ArgoCD and provide valuable insights into how it can enhance your deployment process. Click below to access the LinkedIn post and gain a quick understanding of why ArgoCD is a valuable tool for your software development and deployment needs

Requirements

• Installed kubectl command-line tool.

• Have a Kubernetes cluster and a kubeconfig file. The default location for the kubeconfig file is ~/.kube/config. If you don't have a Kubernetes cluster set up, you can follow this guide to quickly bootstrap Minikube.

• A GitHub account.

Setting Up Continuous Integration (CI) Using GitHub Actions

For this activity, we will use a simple web application written in Python and utilizing Flask. The application has been specifically designed with cloud demonstrations and containers in mind.

To obtain the application code, you can fork this Github repository to your own Github account and then clone it to your local machine to start making changes and customizations as needed.

To create the workflows instruction for GitHub Actions, you'll need to create a YAML file following a specific structure. Start by creating a file named main.yml inside the .github/workflows directory of your repository. This file will serve as the configuration file for workflows. By following this standardized structure, you'll be able to define and customize the actions, triggers, and steps that make up your CI/CD pipeline.

Let's start the workflow configuration with the following structure:

name: ArgoCD demo Build

on:
  push:
    branches:
      - "main"
  pull_request:

In this configuration, we've named the workflow as "ArgoCD Demo Build". It will be triggered on both push events to the "main" branch and pull requests. The workflow will run on an "ubuntu-latest" virtual machine. This setup forms the foundation of the workflow.

jobs:
  test:
    name: 'Test'
    runs-on: ubuntu-latest
    steps:
      - 
        name: Checkout
        uses: actions/checkout@v2

      -
        name: Run tests
        run: make test

Above, we define a job called "Test" that will run on the latest Ubuntu environment (ubuntu-latest).

1. The "Checkout" step ensures that the repository's code is available by using the actions/checkout@v2 action.

2. The "Run tests" step executes the command make test to run the tests.

  build:  
    name: 'Build & Push to Docker Hub'
    runs-on: ubuntu-latest
    needs: test
    steps:
      - 
        name: Checkout
        uses: actions/checkout@v2

      -
        name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      -
        name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ secrets.DOCKERHUB_USERNAME }}/image-name:tag

The next job is "Build & Push to Docker Hub," which also runs on the ubuntu-latest environment.

1. The "Checkout" step ensures that the repository's code is available by using the actions/checkout@v2 action.

2. The "Login to Docker Hub" step authenticates with Docker Hub using the credentials that should be defined in the repository secrets in the GitHub repository settings.

3. The "Set up Docker Buildx" step uses the docker/setup-buildx-action@v2 action to set up Docker Buildx for building the Docker image.

4. Finally, the "Build and push" step uses the docker/build-push-action@v4 action to build the Docker image based on the specified Dockerfile and push it to Docker Hub. Make sure to modify the tags field to match your desired image name and version. And also add credentials to the repository secret before moving on.

Once everything is in place, you can initiate the workflow by pushing your changes to the repository. This action will automatically trigger the workflow to start. To monitor and gain insights into the workflow execution, navigate to the "Actions" tab in GitHub. Here, you'll be able to view the workflow status, check the progress of each step, and identify any errors encountered. If any issues arise, carefully review the error messages provided and make the necessary fixes before proceeding to the next part, which involves setting up the continuous deployment (CD) using ArgoCD.

Setting Up Continuous Deployment (CD) with ArgoCD

In this section, we will explore the process of setting up continuous deployment (CD) using ArgoCD. Building upon the foundation of continuous integration (CI) we established earlier with GitHub Actions, we will now focus on automating the deployment of our application to Kubernetes cluster.

You have the flexibility to utilize any Kubernetes (k8s) cluster at your disposal, whether it's a cloud-based cluster, a bare-metal setup, or even local environments such as Minikube or MicroK8s. ArgoCD is compatible with various Kubernetes configurations, allowing you to seamlessly integrate it into your preferred infrastructure. This versatility enables you to leverage your existing infrastructure or choose a setup that best suits your needs for continuous deployment (CD) with ArgoCD.

To proceed further, we will be utilizing Minikube for our setup. Minikube provides a convenient and lightweight way to run a single-node Kubernetes cluster locally.

Now, let's proceed with the installation of ArgoCD. We will walk through the steps to set up ArgoCD on your chosen Kubernetes cluster, in this case, Minikube.

Installing ArgoCD

To install ArgoCD on your Kubernetes cluster, execute the following commands:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The first command creates a namespace called "argocd" where ArgoCD will be installed. The second command applies the ArgoCD installation manifest, which can be accessed from the official ArgoCD GitHub repository. By executing these commands, you will initiate the installation process and set up ArgoCD within your cluster.

Once the installation is completed, you can verify the installation status by running the following command:

kubectl get nodes -n argocd

This command will display the nodes in the "argocd" namespace, confirming that ArgoCD is successfully installed.

To access the ArgoCD web interface, you can use kubectl port-forwarding to connect to the API server. Execute the following command:

kubectl port-forward svc/argocd-server -n argocd 8080:443

This command will create a port-forwarding tunnel, allowing you to access the ArgoCD UI locally at https://localhost:8080. Simply open a web browser and navigate to the provided URL to access the ArgoCD interface.

To log in to the ArgoCD UI, you will need to retrieve the password from the argocd-initial-admin-secret secret. Follow these steps:

1. Retrieve the secret by executing the following command:

kubectl get secret argocd-initial-admin-secret -n argocd -o yaml

1. The output will include a field called data, which contains the base64-encoded password. Copy the value associated with the password key.

2. Decode the password using the echo and base64 commands. Replace encodedpassword in the command below with the copied value:

echo encodedpassword | base64 --decode

1. The decoded password will be displayed in the terminal. Copy the password string.

2. Return to the ArgoCD UI login page. Enter admin as the username and paste the decoded password into the password field.

Currently, ArgoCD is empty as we haven't configured any applications yet. Let's proceed with configuring ArgoCD to connect to a GitHub repository where our deployment files will be hosted.

It's important to note that in best practices, it is recommended to separate the application repository from the deployment repository. However, for the purpose of this activity, we will keep the deployment files alongside the application files. Please keep in mind that this is not a recommended practice for production-ready environments. In such scenarios, it is crucial to separate the two repositories to ensure a more organized and manageable deployment workflow.

Configuring ArgoCD

To configure ArgoCD to connect to your GitHub repository and deploy your application,

1. Create a YAML file, such as argocd-config.yaml, and add the following content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-cd-demo
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/davWK/argoCD-demo.git
    targetRevision: HEAD
    path: deploy/kubernetes/
  destination: 
    server: https://kubernetes.default.svc
    namespace: demo-app-for-argo-cd

  syncPolicy:
    automated:
      selfHeal: true
      prune: true

Now, let's break down what each section of the YAML file does:

• metadata: Specifies the metadata for the ArgoCD application, including its name and namespace.

• spec.project: Specifies the project within ArgoCD where the application belongs. In this case, it is set to the default project.

• source: Defines the source repository details:

  • repoURL: Specifies the URL of the GitHub repository where your application's deployment files are hosted.

  • targetRevision: Specifies the target revision of the repository to deploy. Here, it is set to HEAD, meaning the latest revision.

  • path: Specifies the path within the repository where your application's Kubernetes deployment files are located. the path ArgoCD will track for any modification

• destination: Specifies the destination details for the deployment:

  • server: Specifies the URL of the Kubernetes API server. Here, it is set to https://kubernetes.default.svc. It can be an external cluster

  • namespace: Specifies the target namespace in which the application will be deployed. In this case, it is set to demo-app-for-argo-cd.

• syncPolicy: Defines the synchronization policy for the application:

  • automated: Specifies that the synchronization should be automated, enabling self-healing and pruning capabilities.

    • selfHeal: Enables self-healing, ensuring the application stays in the desired state.

    • prune: Enables pruning, removing any resources that are no longer defined in the deployment files.

1. Save the file and apply the configuration by running the following command:

kubectl apply -n argocd -f argocd-config.yaml

By applying this configuration, ArgoCD will establish a connection to the specified GitHub repository, fetch the deployment files from the specified path, and deploy the application to the designated namespace within the Kubernetes cluster.

Once you apply the configuration using the command kubectl apply -n argocd -f argocd-config.yaml, you will no longer need to manually apply any changes to your Kubernetes files. ArgoCD takes over the responsibility of tracking and applying changes automatically.

After the initial deployment, ArgoCD continuously monitors the specified GitHub repository and the Kubernetes files within it. Whenever there is a change detected in the repository, ArgoCD will automatically apply those changes to your Kubernetes cluster. This ensures that your application remains up-to-date with the latest version defined in the repository.

With ArgoCD in place, you can focus on making changes to your application's deployment files in the repository, and ArgoCD will handle the synchronization and deployment to the Kubernetes cluster for you. This simplifies the deployment process and provides a seamless experience for maintaining the desired state of your applications.

Writing the python app deployment file for kubernetes

At this stage, the configuration will be created in ArgoCD, but no application pods or services will be available. This is because we have not yet defined the Kubernetes deployment manifest that contains the deployment information for our Python demo app. However, once this manifest is in place, ArgoCD will automatically apply it, resulting in the deployment of the application.

To proceed, you need to create the Kubernetes deployment manifest file that describes the desired state of your application, such as the container image, ports, and any other necessary configurations. Once you have the deployment manifest ready, commit and push it to your GitHub repository.

ArgoCD will then detect the changes in the repository and automatically apply the deployment manifest, triggering the creation of the corresponding pods and services. This automatic synchronization ensures that the deployed application aligns with the desired state defined in the deployment manifest.

To proceed with defining the Kubernetes deployment manifest for the Python demo app:

1. Inside the deploy/kubernetes directory, create a new deployment.yaml.

2. Open the deployment.yaml file and add the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app-deployment
  labels:
    app: python-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: image-name
        image: imageurl
        ports:
        - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: python-app-service
spec:
  type: NodePort
  selector:
    app: python-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000
      nodePort: 30000

1. Save the file.

This deployment manifest defines a Kubernetes Deployment and Service for the Python app. It specifies the container image, ports, replicas, and other necessary configurations.

• The Deployment creates three replicas of the Python app pods.

• The Service exposes the app using a NodePort type, making it accessible on port 30000 of the cluster nodes.

Commit and push the deployment.yaml file to your GitHub repository. ArgoCD will automatically detect the changes and apply the deployment manifest, leading to the creation of the Python app deployment and service.

Once the synchronization is complete, you should see the app pods running and the service available for access.

To access the deployed Python app:

1. Run the following command to get the service information:

kubectl get svc -n <namespace for the Python app>

Replace <namespace for the Python app> with the actual namespace where your Python app is deployed. This command will provide you with the details of the service, including its name, type, cluster IP, and port.

1. Once you have the service information, run the following command to set up port forwarding:

kubectl port-forward -n <namespace for the Python app> svc/python-app-service 8083:<service port>

Replace <namespace for the Python app> with the actual namespace where your Python app is deployed, and <service port> with the port number specified in your service configuration (e.g., 80).

This command establishes a connection between your local machine and the Python app service running in the Kubernetes cluster. It forwards traffic from your local port 8083 to the specified service port.

1. Now, you can access the deployed Python app by opening a web browser and navigating to http://localhost:8083. This will direct your requests to the Python app service running in the Kubernetes cluster.

Conclusion

In conclusion, setting up Continuous Integration (CI) and Continuous Deployment (CD) processes are crucial for efficient software development and deployment. In this article, we explored the steps to configure CI using GitHub Actions and CD with ArgoCD. By integrating these tools into your workflow, you can automate the build, test, and deployment processes, leading to faster and more reliable software delivery.

To learn more about ArgoCD and its capabilities, you can refer to the official ArgoCD documentation available here. The documentation provides comprehensive information, including installation guides, usage examples, and advanced configurations.

For a practical demonstration and understanding of ArgoCD, you can watch the "ArgoCD tutorial" on YouTube by TechWorld with Nana.

To grasp the concept of GitHub Actions and its integration with CI/CD processes, you can watch the "GitHub Action Tutorial" video by TechWorld with Nana. This video explains the fundamentals and basic concepts of GitHub Actions.

Thanks for reading I hope you found the information helpful and informative. If you have any questions or comments, please feel free to reach out to me or leave a comment below.

As you already know, to stay true to the spirit of the challenge and to keep it challenging, this article will not be a detailed step-by-step implementation guide, but rather a post about steps, decisions made, and how challenges were overcome. You can still refer to the documentation on github for something more or less technical, but it is still highly recommended to do everything from scratch to learn properly.

So, the remaining steps were the extra credit: Package Everything in Helm, Implement Persistent Storage, and Implement CI/CD Pipeline. Let's see how I tackled them.

Package Everything in Helm

As I mentioned earlier, at this stage my challenge was clear: I HAD TO LEARN HELM, not just what was needed for the challenge but also to go a bit into the details, in order to get a global and deep understanding of the thing in order to know how to effectively guide my choices in the implementation.

My Helm directory is therefore presented as follows:

Basicly at the root we have the files:

• values.yaml is a file in a Helm chart directory that allows you to set the values of configurable parameters in your Helm chart. It's a way to provide default configuration values for a chart. These values can be overridden by users during the installation of the chart or when upgrading a release.

• chart.yaml is a mandatory file in a Helm chart directory. It contains basic information about the chart.

• And the 'templates' directory contains files that will generate Kubernetes manifest files when the chart is installed. It contains the actual Kubernetes manifests with placeholders that will be populated by the values set in the values.yaml file, if these values are not overwritten during installation or upgrade.

You can visit the official Helm documentation to go into depth, or quickly understand the concepts with this TWN tutorial.

Implement Persistent Storage

The goal here is to persist the data stored in the database and the changes made, so that if the deployments are deleted or redeployed, the data is still available.

To achieve this, I needed to create a Kubernetes PersistentVolumeClaim resource and associate it with the database deployment to mount the database's data storage location (/var/lib/mysql). This is where the database data and the data loaded by the initialization script are stored.

While this can be done without much hassle in an interactive or manual way (e.g., using kubectl apply), the challenge arises when installing with Helm. For the data to be loaded onto this persistent volume, the PVC must be deployed first. Otherwise, there will be errors and the pods will be stuck and won't reach the Ready state. To address this, I utilized a Helm feature called Helm hooks.

Helm hooks are a mechanism to intervene at certain points in a release's lifecycle. They allow you to provide Helm with instructions to perform specific operations based on Helm's lifecycle events, such as install, upgrade, delete, etc. I leveraged this in my case to specify that this PersistentVolumeClaim should be created before the Helm chart is installed or upgraded.

...
metadata:
  name: mariadb-pvc
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
...

• "helm.sh/hook": pre-install,pre-upgrade means that this resource will be managed separately from the rest of the chart's resources. It will be created before the rest of the chart's resources during an install or upgrade.

• "helm.sh/hook-weight": "0" is used to control the execution order of hooks. Hooks with lower weights will be executed before hooks with higher weights. In this case, the PersistentVolumeClaim will be one of the first resources created, as it has a weight of 0.

Implement CI/CD Pipeline

Let's talk about the most fun part, shall we? The roller coaster of CI/CD, haha! Those endless failed jobs before finally seeing a glimmer of hope in the form of a green status, which was only achieved by painstakingly commenting out parts of the workflow file to troubleshoot each step individually. And then, as soon as one step is fixed, the red errors start popping up again, and the cycle repeats until all the steps are fixed and we have a functional pipeline. (It's quite exhilarating, really.)

Enough of that. When setting up this pipeline with GitHub Actions, I wanted to adopt a new Google Cloud security feature to authenticate my workflow when it accesses GCP resources. Previously, we used service account keys, which, if compromised, could put our GCP resources at risk. This is because with service account keys, there's no way to verify who or what is using the keys. So, anyone can access the resources if they get their hands on the keys.

To address this, Google Cloud has introduced Workload Identity Federation, a more secure way to grant access to GCP resources from outside GCP. Workload Identity Federation allows you to authenticate the entity that will use the service account to access the resources. And in this case, the tokens generated are not long-lived but rather short-lived, just for the duration of the operation. So, even if the token is leaked, it will not be valid for accessing GCP resources.

For setting up Workload Identity Federation, refer to this.

Project's Github repository

The first phase is the certification phase, or as I interpret it, the phase of acquiring knowledge and skills, because the goal is to ensure you have a solid understanding of Kubernetes concepts and practical experience. For this purpose, it is recommended in the challenge to complete the Certified Kubernetes Application Developer (CKAD) course by KodeKloud. Personally, I have already been playing with Kubernetes for a while. Therefore, I started the challenge directly. Apart from what is recommended here to gain an understanding of Kubernetes concepts and practical experience, I personally recommend Techworld with Nana's Kubernetes crash course. I really appreciate the clarity and the ability that Nana has to easily digest and present complex topics in an interesting way. Take a look at her crash course.

Now, the hands-on part begins with containerization, containerizing both the application and the database. Containerizing the application is fairly straightforward by following the instructions provided on the CRC website. However, where one needs to focus attention is on interfacing with the database. When dealing with the database, there's no need for containerization since the official Docker image of MariaDB is already available and ready to use. The minor adjustment required here is to load the data into the database using the initialization script via a Kubernetes ConfigMap. This, I believe, can be interactively done through a command resembling something like kubectl create configmap db-init-script --from-file=db-load-script.sql. However, I've developed a habit of configuring almost all Kubernetes resources declaratively. Therefore, I created a manifest for the ConfigMap, not just for this purpose but for all the resources I needed to deploy right from the early stage.

Before exposing the application, it's essential to make the underlying adjustments, namely, initializing the passwords to connect to the database, the connection string to the database, and the service to allow the connection, etc. Therefore, to initialize the root password of the database, I created, of course, a Kubernetes secret containing the base64-encoded value of the password. Then, I referenced it in the environment variables of the MariaDB deployment manifest. It should look something like this in the MariaDB deployment manifest:

env:
  - name: MYSQL_ROOT_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mariadb-root-password
        key: password

This initializes the root password upon launching the database. Then, to establish communication between the application and the database, it's necessary to create the MariaDB service and add it, along with the service and the secret, to the environment variables of the application so that this information is used to initiate the connection to the database when the application starts. It will look something like this:

env:
  - name: DB_HOST
    value: mariadb-service
  - name: DB_USER
    value: root
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mariadb-root-password
        key: password

Next, to expose the application on the internet, I utilized a Load Balancer service type that I associated with the deployment. In my case, the Kubernetes environment I used is GKE on Google Cloud. Therefore, I needed to reserve a static public IP address, which I used for the Load Balancer. Then, I attached the service to the application using the app selector. Once all of this was applied, the application was ready to serve on the internet.

Next comes the second chunk of the project. The first part of this second chunk involved setting up the feature toggle using a ConfigMap. I'll explain how I proceeded. The goal of the feature toggle is to activate the dark mode on the site, so it was necessary to create another CSS file for the dark mode. Then, in the PHP code, I created a condition stating that if the environment variable FEATURE_DARK_MODE = true, then to use the dark mode CSS file, otherwise to use the default file. To make this environment variable value known to the application, I created the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-toggle-config
data:
  FEATURE_DARK_MODE: "true"

And referenced it in the application deployment manifest in the environment section:

env:
  - name: FEATURE_DARK_MODE
    valueFrom:
      configMapKeyRef:
        name: feature-toggle-config
        key: FEATURE_DARK_MODE

With this, once the ConfigMap was applied, the dark mode was activated on the site. Then, it was about manually scaling the application by increasing the number of replicas, and then performing a rolling update with a new version of the application, including a promotion banner on the site, and finally rolling back to the initial state, all as indicated in the instructions on the CRC website.

Now, it was about implementing autoscaling so that the pods scale up to a maximum of 10 if their CPU usage exceeds 50%. I created an autoscaling resource manifest including the autoscaling requirements. Of course, this can be done interactively as indicated in the CRC guide, but as I mentioned earlier, I prefer to set up my configs declaratively whenever possible to benefit from reusability. To simulate the load, I used the lightweight and easy-to-use load testing and benchmarking tool Siege.

Now place to Liveness and Readiness Probes which are Kubernetes mechanisms designed to check the health of a Pod.

1. Liveness Probe: This probe verifies whether your application is running properly. If the liveness probe fails, Kubernetes will terminate the Pod and create a new one as a replacement. This feature is particularly useful if your application has deadlocked and cannot recover without a restart.

   In my configuration, the liveness probe is configured to perform an HTTP GET request to the /live.php endpoint on port 80 of my pod. I have created another PHP file (live.php) in the same location as index.php.

<?php
http_response_code(200);
echo "Application is running";

The probe will start 15 seconds after the Pod initializes (initialDelaySeconds) and repeat every 20 seconds (periodSeconds). It checks whether a 200 response is returned; if not, the Pod will be terminated and recreated.

2. Readiness Probe: This probe checks whether your application is ready to handle incoming traffic. If the readiness probe fails, Kubernetes will halt traffic to the Pod until it becomes ready.

   In my configuration, the readiness probe is configured to perform an HTTP GET request to the /status.php endpoint on port 80 of my Pod. I have also created another PHP file (status.php).

<?php
$dbHost = getenv('DB_HOST');
$dbUser = getenv('DB_USER');
$dbPassword = getenv('DB_PASSWORD');
$dbName = getenv('DB_NAME');

$link = mysqli_connect($dbHost, $dbUser, $dbPassword, $dbName);

if ($link) {
    $res = mysqli_query($link, "SELECT * FROM products LIMIT 1;");
    if ($res) {
        http_response_code(200);
        echo "Application is healthy";
    } else {
        http_response_code(500);
        echo "Application is not healthy: unable to query the database";
    }
} else {
    http_response_code(500);
    echo "Application is not healthy: unable to connect to the database";
}

The probe will start 5 seconds after the Pod initializes (initialDelaySeconds) and repeat every 10 seconds (periodSeconds). This probe goes further by testing the application's connection to the database.

Explore the GitHub repository to delve into manifests and code files: GitHub Repo

To be continued ...

Currently, I'm delving into the extra credit part, which includes Package Everything in Helm, Implement Persistent Storage, and Implement CI/CD Pipeline. Stay tuned for the next updates!

Hello and welcome to this article in a journey of migrating a legacy-built app to the cloud. In this section, we will focus on three aspects: interfacing the application with Cloud Firestore, automating deployment, and exploring how Binary Authorization can reinforce supply chain security while aligning with security policies.

If you're joining us midway, I encourage you to take a look at the previous articles to get up to speed. Otherwise, let's dive in! 😊

integrating the app with Cloud Firestore

Our previous code interacted with MongoDB. With the migration to Google Cloud, we are transitioning away from MongoDB in favor of Firestore, which is Google Cloud's managed NoSQL document database built for automatic scaling, high performance, and ease of application development. To achieve this, we'll need to make modifications to our code, ensuring that our application seamlessly integrates and functions with Firestore.

We will replace the old MongoDB code with the following Firestore integration:

Old:

from pymongo import MongoClient
from bson.objectid import ObjectId
import mongomock
...
if os.environ.get('TESTING'):
    client = mongomock.MongoClient()
else:
    client = MongoClient(os.environ['MONGO_URI'])
db = client.flask_db
todos = db.todos

New:

from google.auth import compute_engine
from google.cloud import firestore
...
credentials = compute_engine.Credentials()
db = firestore.Client(credentials=credentials)
todos = db.collection('todos')

1. from google.auth import compute_engine: This line imports the compute_engine module from the google.auth library, which is used for authentication in Google Cloud environments

2. fromgoogle.cloudimport firestore: This line imports the firestore module from the google.cloud library, enabling interaction with Google Cloud Firestore.

3. The compute_engine.Credentials() call retrieves the default credentials provided by Google Cloud in its environment. These credentials are essential for authenticating with Firestore. In a local or non-Google Cloud service environment, you would need to generate a service account key before being able to authenticate with Firestore. However, in our case, since the code will be deployed on Cloud Run, authentication will be handled using the default service account of Cloud Run.

4. todos = db.collection('todos'). Here, we're defining a Firestore collection. Collections are used to organize documents in Firestore.

Data Insertion: When a POST request is made, the new todo item is added to the Firestore collection 'todos' using the add method. The data is stored as a dictionary.

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        content = request.form.get('content')
        degree = request.form.get('degree')
        todos.add({'content': content, 'degree': degree})

Old:

todos.insert_one({'content': content, 'degree': degree})

New:

todos.add({'content': content, 'degree': degree})

This modification reflects the adjustment needed in the code for Firestore, moving from the insert_one method in MongoDB to the add method in Firestore for adding documents.

Data Retrieval: In the new code, we utilize todos.stream() to obtain a stream of documents from the Firestore collection. In the old code, we used todos.find() to get a cursor to the documents in the MongoDB collection.

Old:

all_todos = todos.find()

New:

all_todos = [{'_id': doc.id, **doc.to_dict()} for doc in todos.stream()]

We now use todos.stream() to iterate over documents and convert them to a dictionary format for retrieval. The '_id' field represents the document ID in Firestore.

Data Deletion: In the new code, we employ todos.document(id).delete() to remove a document from the Firestore collection. In the old code, we used todos.delete_one({"_id": ObjectId(id)}) to delete a document from the MongoDB collection.

Old:

todos.delete_one({"_id": ObjectId(id)})

New:

todos.document(id).delete()

The todos.document(id).delete() method is used to delete a specific document by its ID in Firestore.

After all these updates, the new app.py should look like this:

import os
from flask import Flask, render_template, request, url_for, redirect
from google.auth import compute_engine
from google.cloud import firestore

app = Flask(__name__, template_folder='templates')

# Use the default credentials provided by the Cloud Run environment
credentials = compute_engine.Credentials()

# Use these credentials to authenticate with Firestore
db = firestore.Client(credentials=credentials)

todos = db.collection('todos')

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        content = request.form.get('content')
        degree = request.form.get('degree')
        todos.add({'content': content, 'degree': degree})
        return redirect(url_for('index'))

    all_todos = [{'_id': doc.id, **doc.to_dict()} for doc in todos.stream()]
    return render_template('index.html', todos=all_todos)

@app.route('/<id>/delete/', methods=['POST'])
def delete(id):
    todos.document(id).delete()
    return redirect(url_for('index'))

The adjustments include using Firestore methods for data insertion (todos.add()), retrieval (todos.stream()), and deletion (todos.document(id).delete()), along with integrating the appropriate syntax for Firestore operations.

Testing the new code

To ensure the correctness of the new 'app.py' code, we have to update also the testing approach. The tests aim to verify the functionality of critical components, such as data insertion and deletion, within the context of Firestore integration.

import unittest
from unittest.mock import patch, MagicMock
from app import app

class TestApp(unittest.TestCase):
    def setUp(self):
        self.app = app.test_client()

    @patch('app.todos.add')
    def test_index_post(self, mock_add):
        response = self.app.post('/', data={'content': 'Test Todo', 'degree': 'Test Degree'})

        mock_add.assert_called_once_with({'content': 'Test Todo', 'degree': 'Test Degree'})
        self.assertEqual(response.status_code, 302)

    @patch('app.todos.document')
    def test_delete(self, mock_document):
        mock_delete = MagicMock()
        mock_document.return_value.delete = mock_delete

        response = self.app.post('/123/delete/')

        mock_delete.assert_called_once()
        self.assertEqual(response.status_code, 302)

if __name__ == '__main__':
    unittest.main()

Let's delve into the primary components of this testing suite:

1. Data Insertion Test (test_index_post):

   • This test simulates a POST request to the root endpoint ('/') of the application when a new todo item is added.

   • The @patch decorator is utilized to mock the todos.add method, ensuring that actual Firestore interactions are bypassed during testing.

   • The test asserts that the 'add' method is called with the expected data, and the response status code is as expected (302 for a successful redirect).

2. Data Deletion Test (test_delete):

   • This test mimics a POST request to the endpoint for deleting a specific todo item ('//delete/').

   • The @patch decorator is applied to mock the todos.document method, and a MagicMock is used to mock the 'delete' method of the Firestore document.

   • The test verifies that the 'delete' method is called once and asserts the response status code after the deletion operation (302 for a successful redirect).

These tests ensure that data insertion and deletion operations interact seamlessly with Firestore. The use of mocking allows for isolated testing, focusing on specific components without the need for actual Firestore connections during the testing phase.

Now that the test has been added, you can re-run your Cloud Build pipeline to address any potential minor issues before proceeding with the deployment on Cloud Run.

Automating deployment (CD)

To automate the deployment on Cloud Run after building and pushing the image, add the following step to your Cloud Build configuration (cloudbuild.yaml):

# Step 8: Deploy the image to Cloud Run
- id: 'deploy-image'
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'gcloud'
  args:
  - 'run'
  - 'deploy'
  - '$_SERVICE_NAME'
  - '--image'
  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA'
  - '--region'
  - '$_REGION'
  - '--platform'
  - 'managed'
  - '--allow-unauthenticated'
  waitFor: ['push-image']

This segment of the Cloud Build configuration file handles the deployment of the Docker image to Google Cloud Run. Here's a breakdown of each line's purpose:

• id: 'deploy-image': Provides a unique identifier for this step within the Cloud Build configuration file.

• name: 'gcr.io/google.com/cloudsdktool/cloud-sdk': Specifies the Docker image to be used for this step, which, in this case, is the Google Cloud SDK image.

• entrypoint: 'gcloud': Sets the Docker entrypoint to 'gcloud,' the command-line interface for Google Cloud Platform.

• args: A list of arguments passed to the 'gcloud' command.

  • 'run' 'deploy' '$_SERVICE_NAME': Deploys a new revision of the Cloud Run service identified by $_SERVICE_NAME.

  • '--image' '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA': Specifies the Docker image to deploy, located in the designated Google Cloud Artifact Registry repository.

  • '--region' '$_REGION': Specifies the region where the Cloud Run service is deployed.

  • '--platform' 'managed': Indicates that the Cloud Run service uses the fully managed version of Cloud Run.

  • '--allow-unauthenticated': Permits unauthenticated requests to the Cloud Run service.

• waitFor: ['push-image']: Directs Cloud Build to wait for the completion of the 'push-image' step before initiating this step.

• Afterwards, don't forget to update the substitution section of your Cloud Build configuration to reflect the variables used in this new step.

Now, you can push your code to trigger the pipeline. If the pipeline runs successfully, you will obtain the access link for your application. Navigate to the Google Cloud Console, go to Cloud Run, and click on the name of your newly deployed service to retrieve the access URL for your application.

Binary Authorization

The Binary Authorization is a security control mechanism during the deployment of container images on Google Cloud platforms like Cloud Run, GKE, Anthos Service Mesh, and Anthos Clusters. Its primary function is to either authorize or control the deployment of images that have been attested as secure or trusted. This attestation involves subjecting the image to various processes such as testing, vulnerability scanning, and even manual signatures. Only after the image meets predefined conditions is it considered validated and allowed for deployment on the platforms.

Binary Authorization is responsible for defining and enforcing this policy. The controls or checks are executed using attestators, which can be custom-created, fixed, or generated by tools like Cloud Build (currently in preview).

For this project, I explored configuring Binary Authorization using the built-by-cloud-build attestor to deploy only images built by Cloud Build. With a well-crafted and robust Cloud Build configuration (incorporating various tests, vulnerability analyses, etc.), this approach can significantly save time compared to creating and using a custom attestor. However, as of the time of writing this article, Binary Authorization with cloud build attestor is in preview.

The main challenge with using the built-by-cloud-build attestor is that it is generated only once during the build with Cloud Build. This may not align well with continuous delivery (CD), especially for recurrent executions of the CI/CD pipeline. Ideally, with each new run of the pipeline, a new attestor should be generated to update the Binary Authorization policy. This becomes problematic, especially if the Binary Authorization policy is configured at the organization level, impacting all other deployments. From a personal perspective, to address this, it would be beneficial if Cloud Build generates the attestor once and uses it for subsequent pipeline executions. Currently, the custom attestor provides a workaround for this limitation. However, for simplicity, it would be ideal if Cloud Build handles this process seamlessly.

Follow this link for the setup of Binary Authorization."

This concludes the article. Thank you for reading. You can find the configurations and code for this project in the following Git repository.

In this article, we'll be transforming Docker Compose services into Kubernetes objects and deploying them in a Kubernetes environment.

To follow along, you'll need some knowledge of Kubernetes, have completed the lab described in the previous article, or have done something similar, make sure you have a Kubernetes environment ready. As of now, I'm using Digital Ocean's Kubernetes Engine. I mention 'as of now' because if you've been here from the beginning, you're probably aware that our project's ultimate goal isn't just deploying on K8s. It's a journey of migrating a traditional app to a serverless cloud setup. The next step in this series will involve migrating to Google Cloud. Oh, did I forget to mention? I'm all about Google Cloud—I recently even snagged my Professional Cloud Architect certification. So, expect Google Cloud to pop up regularly in my discussions, and the rest of this series will be purely GCP-focused.

Enough chatter, let's dive into the real stuff!

Build the application image and push it to the Docker registry

If you haven't done so already, I invite you to clone our project's repo here. Navigate to the docker folder, where all the Docker-related elements of the project are stored. Explore the content a bit, and once you're ready, come back, and let's continue. If you don't have a Docker Hub account yet, I recommend creating one.

Now, in your terminal, log in with docker login using your Docker Hub account information. After that, build the image, tagging it with your username and the image name.

docker build -t <username>/<image-name> .

Finally, push the image to Docker Hub.

docker push <username>/<image-name>

Export MongoDB data

As part of our migration process, it's crucial to ensure we retain our data. To achieve this, let's export the data stored in the MongoDB container that we'll later use when deploying MongoDB on Kubernetes.

Export the existing MongoDB database from the Docker Compose setup:

1. Access the MongoDB database container shell.

    docker exec -it <mongo_db_service> bash
   

2. Export all data from the MongoDB database.

    mongodump <file name>
   

3. Exit the MongoDB database container shell.

    exit
   

4. Copy the 'dump' folder from the MongoDB container to a specified destination.

    docker cp <mongo_db_service>:/dump <destination>
   

Install MongoDB on Kubernetes

Now, while connected to the Kubernetes cluster, let's install MongoDB using Helm:

helm install mongo-helm oci://registry-1.docker.io/bitnamicharts/mongodb --set auth.rootUser=root,auth.rootPassword="defineYourRootPassword"

This command leverages Helm, a Kubernetes package manager, to install MongoDB from a chart hosted on Docker's registry.

• The part --set auth.rootUser=root,auth.rootPassword="DefineYourPassword" specifies the username and password for the MongoDB root user.

Make sure to save the output of this command; we'll be using it to construct the database connection URI.

Verify that everything is installed correctly with the following commands:

kubectl get pods
kubectl get services

Restore data

It's time to restore the database:

1. Navigate to MongoDB Kubernetes pod

    kubectl exec -it --namespace default mongodb_pod -- /bin/bash
    mongosh
    use admin
    db.auth('root', 'password')
   

2. Create a non-root MongoDB user:

db.createUser({
  user: 'username',
  pwd: 'password',
  roles: [
    { role: 'readWriteAnyDatabase', db: 'admin' },
    { role: 'dbAdminAnyDatabase', db: 'admin' },
    { role: 'clusterAdmin', db: 'admin' }
  ]
})
exit

1. Restore the Docker Compose database dump to the new MongoDB pod:

Copy the database dump folder previously copied into the MongoDB pod:

kubectl cp <mongodb_dump_location_filename> <mongodb_pod>:/tmp/

Navigate to the MongoDB pod shell:

kubectl exec -it --namespace default mongodb_pod -- /bin/bash

Change the directory to the dump directory and list all MongoDB folders to verify the contents:

cd /tmp/dump
ls

Restore the app database:

mongorestore --uri="mongodb://username:password@localhost:27017/?authSource=admin" app_db -d app_db

Here's how it's formed:

• mongodb://: This is the prefix to identify that we're connecting to a MongoDB instance.

• <username>:<password>@: This part specifies the username and password to connect to the MongoDB instance. You would replace <username> and <password> with the actual username and password. In your case, the username is the one created earlier.

• mongo-helm-mongodb.default.svc.cluster.local:27017: This is the host and port where the MongoDB server is running. mongo-helm-mongodb.default.svc.cluster.local is the DNS name for the MongoDB service in your Kubernetes cluster, and 27017 is the default port for MongoDB.

Exit the MongoDB pod shell:

exit

The new MongoDB is now ready for use.

Deploy and connect the application to the database

First, let's create the Kubernetes secret that will contain the connection string for the database. We're using the secret object because our connection string contains sensitive information. Kubernetes provides the secret object precisely for scenarios like this. If it were just configuration information or environment variables, a ConfigMap object would be more suitable.

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque
data:
  mongo-uri: <base64-encoded-mongo-uri>

Create a YAML file and paste this content into it. Name the file as you see fit. Note that the mongodb-uri field under data should contain the base64-encoded MongoDB URI. Replace the placeholder with the actual base64-encoded connection string.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-to-cloud-deployment
  labels:
    app: legacy-to-cloud
spec:
  replicas: 3
  selector:
    matchLabels:
      app: legacy-to-cloud
  template:
    metadata:
      labels:
        app: legacy-to-cloud
    spec:
      containers:
      - name: legacy-to-cloud
        image: docker_username/image_name:tag
        ports:
        - containerPort: 5000
        env:
        - name: MONGO_URI
          valueFrom:
            secretKeyRef:
              name: mongodb-uri-secret
              key: mongodb-uri

---
apiVersion: v1
kind: Service
metadata:
  name: legacy-to-cloud-service
spec:
  type: LoadBalancer
  selector:
    app: legacy-to-cloud
  ports:
  - protocol: TCP
    port: 80
    targetPort: 5000

Create a second YAML file for your application's manifest and paste this content into it. The env section of the container in the Deployment references the MongoDB URI from the secret we created earlier. Ensure that the secret name and key match the values used in the secret manifest. Also, ensure that the selector in the Service matches the one in the Deployment. This is crucial for linking the pods to the service.

If everything looks good, let's proceed with deploying our application. You can use the following command to validate the syntax of your YAML file and perform a dry run:

kubectl apply -f filename.yaml --dry-run=client --validate=true

This command checks the syntax of your YAML file and prints out the resources that would be created or modified without actually applying the changes. If there are any syntax errors, this command will highlight them.

If everything is okay, create the resources with the following command:

kubectl apply -f filename1.yaml -f filename2.yaml

Replace filename1.yaml and filename2.yaml with the actual names of your YAML files.

Get the access IP address with the command:

kubectl get svc

Identify the service for your application and copy its external IP. Paste it into your browser to access the application.

Well, that wraps up this section on the migration to Kubernetes.

A little gift for the road?

Haha, did you know there's a tool to speed things up? Because here, we've created YAML manifests to deploy K8s resources. This deployment is a simple one, but imagine if it were a massive deployment with hundreds of Docker Compose services, unimaginable complexities, etc. Would we sit down and manually create manifests for all that complexity? Of course not :) Enter Kompose. Kompose is a conversion tool for Docker Compose to container orchestrators like Kubernetes. It takes a Docker Compose file and translates it into Kubernetes resources.

Kompose is a handy tool for those familiar with Docker Compose but aiming to deploy their application on Kubernetes. It automates the creation of Kubernetes deployments, services, and other resources based on the services defined in the Docker Compose file.

However, it's worth noting that not all Docker Compose features and options are supported by Kompose, so some manual tweaking of the generated Kubernetes resources might be necessary. Here's an excellent guide that addresses our use case well.

What next?

And that's a wrap for this article! In the next one, we're heading to the GOOGLE CLOUUUUUUUD :) and beginning to introduce DevOps tools and practices to automate and speed up our work. We're talking about stepping up the game. We'll be using Google Cloud DevOps tools—Cloud Build for CI/CD, Artifact Registry for container images, GKE for deployments. Plus, we'll dive into DevSecOps tools and practices, leveraging the security available within the Google Cloud ecosystem.

Thanks for reading, and see you soon in the next article in the series!"

In this article, we will focus on containerizing your app. However, if you're building an app from scratch, that's perfectly fine (in fact, it's even better). For this example, I'm using this DigitalOcean guide to build a simple TODO app using Python (Flask) and MongoDB as the database. I've made some customizations to make it look better, but the main point is to build something that uses a NoSQL document-based database, as this will be required for the upcoming work.

You can clone the repository of the app here on GitHub if you haven't built your own.

Once you have your app built, let's get started!

Dockerfile

Here is the structure of the application directory that we will containerize, followed by the Dockerfile.

.
├── app.py
├── LICENSE
├── README.md
├── requirements.txt
├── static
│   └── style.css
└── templates
    └── index.html

The app.py file is the main application file that contains the Flask app code. The requirements.txt file contains the list of Python dependencies required by the application. The static/ directory contains static files such as CSS, JavaScript, and images. The templates/ directory contains the HTML templates used by the Flask app.

# Use a minimal base image
FROM python:3.9.7-slim-buster AS base

# Create a non-root user
RUN useradd -m -s /bin/bash flaskuser
USER flaskuser

# Set the working directory
WORKDIR /app

# Copy the requirements file and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Add the directory containing the flask command to the PATH
ENV PATH="/home/flaskuser/.local/bin:${PATH}"

# Use a multi-stage build to minimize the size of the image
FROM base AS final

# Copy the app code
COPY app.py .
COPY templates templates/
COPY static static/

# Set environment variables
ENV FLASK_APP=app.py
ENV FLASK_ENV=production

# Expose the port
EXPOSE 5000

# Run the app
CMD ["flask", "run", "--host=0.0.0.0"]

Here's a walkthrough and breakdown of the Dockerfile:

1. The Dockerfile starts with a FROM instruction that specifies the base image to use. In this case, it's python:3.9.7-slim-buster, which is a minimal base image that includes Python 3.9.7 and some essential libraries.

2. The next instruction creates a non-root user named flaskuser using the RUN and useradd commands. This is a security best practice to avoid running the container as the root user.

3. The WORKDIR instruction sets the working directory to /app, which is where the application code will be copied.

4. The COPY instruction copies the requirements.txt file to the container's /app directory.

5. The RUN instruction installs the dependencies listed in requirements.txt using pip. The --no-cache-dir option is used to avoid caching the downloaded packages, which helps to keep the image size small.

6. The ENV instruction adds the directory containing the flask command to the PATH environment variable. This is necessary to run the flask command later.

7. The FROM instruction starts a new build stage using the base image defined earlier. This is a multi-stage build that helps to minimize the size of the final image.

8. The COPY instruction copies the application code (app.py), templates (templates/), and static files (static/) to the container's /app directory.

9. The ENV instruction sets the FLASK_APP and FLASK_ENV environment variables. FLASK_APP specifies the name of the main application file, and FLASK_ENV sets the environment to production.

10. The EXPOSE instruction exposes port 5000, which is the default port used by Flask.

11. The CMD instruction specifies the command to run when the container starts. In this case, it runs the flask run command with the --host=0.0.0.0 option to bind to all network interfaces.

With this Dockerfile, the application can be containerized and executed. However, it's important to note that our app requires a database to store the data created or generated while it's running. Of course, you could separately pull a MongoDB database image and run it independently. Then, make adjustments on both sides to establish communication between the two containers so that the app can successfully store data in the database. While this approach works, it may consume time and be a bit tedious. To streamline the process, we will instead move forward with Docker Compose. In Docker Compose, everything is declared in a YAML file, and by using the docker-compose up command, we can start and operate the different services seamlessly, saving time and effort.

Streamlining Database Integration with Docker Compose

Here is the basic Docker Compose YAML file that we will use to streamline the process.

version: '3.9'

services:
  db:
    image: mongo:4.4.14
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  web:
    build: .
    container_name: "myflaskapp"
    ports:
      - "5000:5000"
    environment:
      - MONGO_URI=mongodb://db:27017
    depends_on:
      - db

volumes:
  mongo-data:

This Docker Compose YAML file is configured to set up two services: a MongoDB database (db) and a web application (web). Here's a breakdown:

• Version: Specifies the version of the Docker Compose file format being used (3.9 in this case).

• Services:

  • Database (db):

    • Uses the MongoDB version 4.4.14 image.

    • Maps the host port 27017 to the container port 27017.

    • Utilizes a volume named mongo-data to persistently store MongoDB data.

  • Web Application (web):

    • Builds the Docker image from the current directory (.).

    • Sets the container name as "myflaskapp."

    • Maps the host port 5000 to the container port 5000.

    • Defines an environment variable MONGO_URI with the value mongodb://db:27017, establishing a connection to the MongoDB service.

    • Specifies a dependency on the db service, ensuring that the database is started before the web service.

• Volumes:

  • Defines a volume named mongo-data for persisting MongoDB data.

In summary, this Docker Compose file orchestrates the deployment of a MongoDB database and a Flask web application, ensuring they can communicate and function together seamlessly.

Now navigate to the directory with the Docker Compose file and run docker-compose up to start MongoDB and a Flask web app. Access the app at http://localhost:5000 to ensure everything works as expected.

To stop, use docker-compose down.

All good? Next up: migrating the workflow to Kubernetes in the next article.

Deploying an application on AWS EKS (Elastic Kubernetes Service) can be a powerful way to ensure scalability and reliability for your application. However, the process can be complex and time-consuming, especially when it comes to ensuring the security and compliance of your deployment. In this article, we'll show you how to simplify the process and ensure your deployment is secure with GitLab CI/CD and Checkov. GitLab CI/CD provides a powerful toolset for automating the deployment process and improving collaboration among team members, while Checkov is a Security as Code tool that can help you automatically scan your configuration files for potential security and compliance issues. By integrating these tools into your deployment pipeline, you can ensure your deployment is secure and compliant with industry best practices, all while saving time and effort.

Prerequisites

Before proceeding you will need the following :

• Set up a GitLab project with runners to execute CI/CD jobs

• A container registry (a docker hub repo is more than enough)

• A running AWS EKS cluster

• Some knowledge of Docker and Kubernetes

The different steps are as follows

1. Set up the application code and the Dockerfile

2. Define CI/CD GitLab variables

3. Set Kubernetes manifest files

4. Set up the CI/CD pipeline

5. Trigger the pipeline with a git push.

Directory structure

├── Dockerfile
├── .gitlab-ci.yml
├── .k8s
│   ├── deployment.yaml
│   └── services.yaml
└── src
    ├── app.py
    └── requirements.txt

The application files and the Kubernetes configurations are respectively in the src and .k8s directories. and the Dockerfile and the GitLab-ci script are at the root of the directory.

Set up the application code and the Dockerfile

Use whatever language or framework you want to create the application you want, the main thing is to have an application that you can containerize with a Dockerfile. Personally, I used a simple Python code that uses the Flask framework to create a web application that displays a "Hello, World!" message.

For the Dockefile, here is an example of what it could look like :

FROM python:3.9-slim-buster

WORKDIR /app

COPY src/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY src/app.py .

CMD ["python", "./app.py"]

I recommend you test your docker image locally before continuing.

Define CI/CD GitLab variables

To connect to AWS, Kubernetes, and Docker Hub from GitLab CI, you need to define variables in the GitLab CI/CD pipeline. You can define these variables in the GitLab project settings under CI/CD > Variables.

To connect to AWS

• ${AWS_ACCESS_KEY_ID}: This variable contains the access key ID for the AWS account used to deploy the application.

• ${AWS_SECRET_ACCESS_KEY}: This variable contains the secret access key for the AWS account used to deploy the application.

• ${AWS_DEFAULT_REGION}: This variable contains the AWS region where the application will be deployed.

The variables related to the Docker hub or container registry

• ${CI_REGISTRY_USER}: This variable contains the username used to authenticate with Container Registry, it can be docker hub, GitLab registry or whatever you want

• ${CI_REGISTRY_PASSWORD}: This variable contains the password used to authenticate with the Container Registry.

• ${CI_REGISTRY_IMAGE}: This variable contains the name of the Docker image in the Container Registry.

• ${CI_REGISTRY_IMAGE_VERSION}: This variable contains the version or tag of the Docker image in the Container Registry.

The configuration file to access Kubernetes

• ${KUBECONFIG}: This variable of type file contains the Kubernetes configuration file used to authenticate with the Kubernetes cluster. this file is available at this path ~/.kube/config so when adding it to Gitlab make sure you choose File as the type

Set Kubernetes manifest files

Now it's time to define manifest files for Kubernetes deployments. As you would have seen above these files are located in the .k8s folder.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: my-app
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}
        resources: {}
status: {}

deployment.yaml defines a Kubernetes Deployment for the application named "my-app". The Deployment creates a single replica of the application and specifies the container image to be used for the application using the variable ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}. This variable references the Docker image built and pushed to a Docker registry.

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: my-app
  name: my-app
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 5000
  selector:
    app: my-app
status:
  loadBalancer: {}

services.yaml defines Kubernetes Service for the "my-app" application. The Service exposes the application on port 80 and routes traffic to the listening port of our hello world app which is port 5000 on the application container. The file also specifies the labels to be used to identify the application in Kubernetes.

Hint: To generate a quick template on which to base and write these configuration files and thus reduce the risk of error and save time, there is an option of the kubectl command that can be very useful.

The --dry-run=client -o yaml option in the kubectl command generates a YAML representation of the Kubernetes resource that would be created or modified without actually creating or modifying the resource. Here is an example of how to generate our yaml file

#deployment
kubectl create deployment <app_name> \
     --image=<the_docker_image> \
     --dry-run=client -o yaml > deployment.yaml

#service
kubectl expose deployment <app_name> \
     --port=80 --target-port=5000 \
     --dry-run=client -o yaml > service.yaml

This should generate the two yaml files that you will adjust to fit your use. You can also use the --dry-run option with the kubectl command to validate a Kubernetes YAML file without actually applying it to a cluster.

To use the --dry-run option to validate a Kubernetes YAML file, you can run the following command:

kubectl apply --dry-run=client -f <yaml_file_path>

Set up the CI/CD pipeline

Now we can start setting up the GitLab script for our pipeline. The script in our case here will have 5 steps.

The script consists of the following stages:

• docker build: Builds the Docker image and tags it with the registry information.

• docker push: Pushes the Docker image to the registry.

• test: Runs the Checkov tool to validate the Kubernetes deployment and service files.

• deploy services: Deploy the Kubernetes services to EKS using the kubectl command.

• deploy the app: Deploys the Kubernetes application to EKS using the kubectl command.

Let's go a little further into the test phase, our test here is much more security oriented. we have integrated Checkov into the pipeline. Checkov is an open-source tool used for static code analysis of infrastructure-as-code (IAC) files. In this case, it will be used to perform security and compliance checks on the Kubernetes YAML files in the .k8s directory.

By running Checkov on the .k8s/deployments.yaml and .k8s/services.yaml files, the GitLab CI/CD pipeline can ensure that the Kubernetes resources being deployed meet the security and compliance requirements defined in the policies and rules being enforced by Checkov.

stages:
  - docker build
  - docker push
  - test
  - deploy services
  - deploy app


Build docker image:
  image: docker:stable
  stage: docker build
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD}
  script:
    - docker build -t the-app .
    - docker tag the-app:latest ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION} 

Push to registry:
  image: docker:stable
  stage: docker push
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD}
  script:
    - docker push ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}

Test:
  image: bridgecrew/checkov:latest
  stage: test
  script:
    - checkov -d .k8s/deployments.yaml
    - checkov -d .k8s/services.yaml
  allow_failure: true



Deploy services on EKS:
  image: ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}
  stage: deploy services
  before_script:
    - export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    - export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    - export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
  script:
    - cd .k8s
    - kubectl --kubeconfig ${KUBECONFIG} apply -f services.yaml
  rules:
    - changes:
      - .k8s/services.yaml


Deploy app on EKS:
  image: ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}
  stage: deploy app
  before_script:
    - export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    - export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    - export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
  script:
    - cd .k8s
    - kubectl --kubeconfig ${KUBECONFIG} apply -f deployment.yaml
    - kubectl --kubeconfig ${KUBECONFIG} rollout status deployments

• Docker build stage: Builds a Docker image for the application specified by the Dockerfile using the official docker:stable image. Before building the image, it logs in to the Docker registry using the username and password provided as GitLab CI environment variables ${CI_REGISTRY_USER} and ${CI_REGISTRY_PASSWORD}. After building the image, it tags it with ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}.

• Docker push stage: Pushes the Docker image created in the previous stage to the GitLab CI registry using the docker push command.

• Test stage: Runs checkov tool for Kubernetes manifest files deployment.yaml and services.yaml. in this case, this stage is allowed to fail and does not prevent the pipeline from continuing.

• Deploy services on EKS stage: Deploys the Kubernetes services specified in the services.yaml file to the Amazon Elastic Kubernetes Service (EKS) cluster. Before deploying, it sets the AWS credentials and region environment variables. It only runs the deployment if the services.yaml file has been modified since the last pipeline run.

• Deploy app on EKS stage: Deploys the Kubernetes deployment specified in the deployment.yaml file to the EKS cluster. Before deploying, it sets the AWS credentials and region environment variables. After deploying, it checks the rollout status of the deployment.

To make sure that this script is valid, you can use a lint tool to validate the script, it will help to quickly fix the errors and save time. For my part, I used glab CLI tool which with the command glab ci lint allows validating the script to make sure that everything is correct.

Trigger the pipeline with a git push.

To trigger this GitLab CI pipeline, you need to commit and push the code changes to the GitLab repository that contains this GitLab CI script.

Once you have pushed the changes to the repository, GitLab CI automatically detects the changes and starts running the pipeline. The pipeline can also be triggered manually by clicking the "CI/CD" tab in the GitLab repository and clicking the "Run Pipeline" button.

Note that to run the pipeline successfully, you need to ensure that you have configured the necessary environment variables on GitLab CI, such as CI_REGISTRY_USER, CI_REGISTRY_PASSWORD, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, and KUBECONFIG. These variables are used to log in to the GitLab registry, authenticate with AWS, and connect to the Kubernetes cluster.

You can find the files I used here on my GitHub or GitLab

I remain open to any contribution and suggestion to improve my work. Do not hesitate to let me know your contribution or suggestion by opening an issue.

Thanks for reading :)

It is better if you are new to all this Cloud related stuff, to take the AWS Cloud Practitioner certification exam, as recommended in the CRC guide as a first step. If you are not familiar with cloud environments or come from another cloud provider, taking this exam will help you validate your knowledge of the cloud and your familiarity with the different services of AWS (in case you come from another cloud provider).

Personally, I had to quickly complete the AWS Cloud Practitioner Quest to refresh my knowledge of AWS as I am quite familiar with the cloud and regarding AWS, I have already worked with some services. The quest is free; you can try it here.

Well now it's good for the intro, let's get to the heart of the matter

Big picture of deployments

This project is about creating a website which is hosted on amazon S3, then on the site display the number of visitors which is calculated with an AWS Lambda Function and stored in DynamoDB table, then automate all process, Website publication and the resources deployment via a CI/CD pipeline. My resume page is available https://aws.davidwoglo.me/

Website

The first steps of the project consist in setting up a website, using a completely different approach than the traditional one. In a traditional way, we set up a machine or VM on which we install a web server (Apache, Nginx, or whatever you choose) then we upload the html/CSS files etc. and we proceed to some configuration to make the site available.

There I'm spared this server preparation task (and the underlying configs). I just uploaded the website files to Amazon S3 bucket, the object storage service of AWS, the equivalent of Cloud Storage at Google Cloud, then I did some small configuration in two three clicks, just to let S3 know that I want to use this bucket to host a static website, and then we have a ready to use website. No need for a physical server or additional tools to install. This is the serverless approach, and this is what the rest of the project is based on, I didn't need to use any server or VM.

In order for my site to be accessible via a user friendly and secure HTTPS URL, it is necessary to manage the DNS and the ssl certificate configuration, so I used AWS Certificate Manager to obtain a certificate for my domain which ownership had to be verified by automatic mail, due to some problems related to my Custom Domain provider but it is recommended to use a CNAME record to do that. Then to route the DNS traffic I used Amazon Route 53, and the distribution of website content is sped up by Amazon CloudFront(CDN Service). All these configurations were done manually in a separate way and tied together at the end to make things work.

At this point let's make a small comparison with how Google Cloud handles it. At Google Cloud all this can be included in the creation of a Load Balancer where we will just have to activate the automatic management of SSL for HTTPS and CDN for content caching.

Counting website visitors

My web page include a visitor counter that displays how many people have accessed the site.To do this, I created an AWS Lambda function, a DynamoDB table and a REST API. On one side, I wrote a python code that is executed by Lambda, the python function is to get the current number of visitors stored in DYnamoDB and increment it by 1 each time a visitor access my page, on the other side, I added a javaScript code to the files of my site. The job of this script is to get the number of visitors present in the Dynamo table and display it on my page, the communication between the JS code and the database is done via a REST API that I set up using Amazon API Gateway.

This is the part that gave me headaches when I was doing the project on Google Cloud. I didn't use an API Gateway there (because honestly, I didn't know), so I used an open source Functions Framework for Python in which I used client library API to communicate with Cloud Firestore which is the equivalent of DynamoDB.

Automation (CI/CD, IaC, Source Control)

To accelerate and simplify the update of my deployments, whether it is the website (frontend) or the underlying resources (backend), I need to set up a CI/CD pipeline. A CI/CD pipeline is a series of steps that must be performed in order to deliver a new version of software. Well for the website, it's ok, we can manage it as software, (it's composed of files as a software is), the question you could ask yourself is : What about the cloud resources in the background ? Since they are not files, right? This is where Infrastructure as Code (IaC) comes in. But before talking about it, let's see how the CI/CD for the front end was set up. I created a source control repository on Github where I put the website files, then I wrote a workflow file that instructs Github Action on how to update my website every time I make a push. Now let's talk about the Infrastructure as Code stuff, i.e. how to manage provision resources through machine-readable definition files, rather than use an interactive configuration as it is traditionally done. Well I used Terraform to define the DynamoDB table, the API Gateway, the Lambda function configurations in a template and deploy them with Terraform CLI. You see, now that we can also manage our infrastructure as a software, we can integrate it in a CICD to accelerate the process of deployment and update of the infrastructure resources. I proceeded in the same way as the website to set up the Backend pipeline, just that here the Github Actions workflow file is a bit more complex.

You can access my frontend repository here and the backend one here

Well, here is how things went during this project. Thanks for reading :) Please check below for some useful resources.

Useful resources

• Use an Amazon CloudFront distribution to serve a static website
• What is DNS? | How DNS works
• Serverless Land
• DynamoDB Guide
• API Projects
• Python
• Amazon API Gateway CORS Configurator
• Java Script
• API Calls
• Automate Terraform with GitHub Actions

It all started after I passed my certification. I thought to myself, this is it, you've achieved your first short term goal of getting into the Cloud world, something you've been studying tirelessly for in recent months. But I must admit I thought that once certified the offers would fall left and right. Haha !!! it's not the case and it wasn't going to happen, we can't make you an offer like that just because you are certified inh, Certified people, we can find all over the place lately. Certification at best is just the key to open the door of Cloud house 😀, it just offers the opportunity to be consulted, to have a little bit of visibility or I would say consideration. But to get a job (like settling down in the Cloud house 😀), to get offers as I thought, you have to justify your know-how and your knowledge in the Cloud by practical, relevant and convincing experiences. And I can see from afar this question that we juniors often ask ourselves "How can we get experience if we are not given the opportunity to join a company where we can develop these experiences? 🤔" Honestly (at least for the IT world) we can do without it, we can justify our skills, our experiences, our knowledge by projects, challenges, that make us go through the pitfalls, the mistakes, the solutions, the obstacles, the dark days, the light that make us build and help us to polish a profile well adapted and that constitutes a solution to the needs and problems of companies. It is with this in mind that after watching a video of Forrest Brazeal, I came across the Cloud Resume Challenge.

What is The Cloud Resume Challenge ?

The Cloud Resume Challenge is a hands-on project designed to help bridge the gap from Cloud certification to Cloud job. It incorporates many of the skills that real Cloud and DevOps engineers use in their daily work. The Cloud Resume Challenge is a multiple-step (It's roughly 16 steps) resume project, from the creation of a website to the implementation of a CI/CD pipeline, which helps build and demonstrate skills fundamental to pursuing a career as a Cloud Engineer.

Certification

The first step of the challenge is to get a Cloud certification of beginner or associate level. I don't think it's an obligation, but from my personal point of view I strongly recommend it, especially when you are a beginner or if you have a non-technical background. it will allow you to acquire and validate the fundamentals that you need to pursue a career in the cloud. For me, I had to pass the associate level certification of Google Cloud, the Google Cloud Associate Cloud engineer thanks to the Google Africa Developer ScholarshipGADS 2021 program

The architecture

Website

Setup

To go fast and get to the heart of things, I didn't have to write HTML files from scratch, I rather took a free template that I modified and adapted to my needs with the little and old knowledge I have in HTML and CSS. Once the website files are ready, it must be hosted, right? Since the challenge is based on a serverless spirit, the site is not hosted on a server but on the Object storage service of Google Cloud. By uploading website contents as files to Cloud Storage, we can host static websites on buckets. See how to do it.

Make the site accessible via a domain name and secure it

Usually to make a site accessible via a domain name, you must map the IP address of the server hosting the site to a domain name. But in our case there is no server in play that has an IP address, what can I do? Fortunately we have the managed HTTP Load Balancer on Google Cloud which can allow us to set up a LoadBalancer with a public IP address and point this address to a Bucket that hosts a static site. This solves the problem of IP address, we now have an IP address to access our website, we can now link a domain name to this IP address and access our website as usual. But with only that, the site is only accessible in HTTP, for more security the site must use HTTPS, again thanks to Google Cloud's LB we can set this up, without too much trouble, Google Cloud's HTTPS LB has an option for automatic management of SSL certificates that can be activated during the implementation of the LB.

But in a corporate environment it is better that the certificates are managed and controlled by you For the domain name I used Namecheap, however you can use Cloud DNS from Google Cloud to do this

Website visitors count

This part of the challenge consists in having the number of visitors who have visited the site. For this I wrote a JavaScript code in addition to the website files on one side, and on the other side a function in python which is hosted on Google Cloud Function. The purpose of the javascript is to call the execution of the python function which in turn is executed, calculates and stores the result in the Document NoSQL database, Firestore and returns the total number of visitors in Json format to the JS code which is responsible for displaying it on the site, This operation is done each time there is a visit to the page. The python function in this case serves as an API to avoid having the JS code communicate directly with the database

Test

To ensure that the python code works as it should and returns the expected result, it was necessary to write python a test using the unittest module provided by the Python standard library which help write and run tests for Python code. Since the JS code needs to get the data concerning the number of visitors in json format, the python test here verifies if the data returned by the function are indeed in json format

Well I think we can take a coffee break ☕

So far everything that has been done has been manually clicking around in the Google Cloud console. In the following stages, the methods and techniques used by DevOps Engineers will intervene, in particular Infrastructure as Code, GitOps and the CI/CD (I admit it's not that hard in this challenge but it's the same operating mode)

Infrastructure as Code

Here I discovered a rather brilliant thing. Before telling you, know that it is planned here to define resources in a Terraform template and deploy them using the Terraform CLI. But instead of getting into another hassle of rewriting everything I've done so far in Terraform, testing it, destroying it, aligning it to what exists and all that, I discovered a Google Cloud tool that I can use to generate Terraform code for resources in a project, folder, or organization. The gcloud beta resource-config bulk-export --resource-format=Terraform command exports resources currently configured in the project, folder, or organization and prints them to the screen in HCL code format. (Of course that's what I did) So what's left for me to do is manage these Terraform templates with a source control system, so I can integrate it into the pipeline later. aSo what's left for me to do is manage those templates with Github, so I can integrate it into the pipeline later. and when I need to make changes to my Cloud resources , I just have to adjust a few lines and that's it.

But I point out, before using this tool, it is necessary to already have Terraform bases. Here's a great guide to getting started

CI/CD

To setup the CI/CD pipeline or I would rather say the pipelines, since there are two, one for the frontend and another for the backend. I created a Github repository for the frontend (i.e. the website files), and thanks to Cloud Build I automate the update of the website which is triggered each time I make a push. On the other side for the Backend (i.e. the infrastructure resources that are defined in a Terraform template), I also created a Github repo for this purpose. So when I make changes to .tf files my Cloud resources are automatically updated, whether it's deletion, change or addition, all of this is done automatically without any intervention on my part.

In the end ...

Well, that's an overview of how I managed to complete this challenge, which by the way is very rewarding, allowed me to learn and discover really useful things, to identify my weaknesses and how to strengthen them and above all has allowed me to have relevant experience in the cloud. and I don't intend to stop there 😉 Of course some of the steps were not fun, but thanks to the help of some of my SWE connections, in particular Vincent Bakpatina, Corneille ALLOGBALO, Abdel-Khafid ATOKOU and Samtou Assekouda I was able to overcome.

Here are some useful resources 👇🏾

Here is my Github repo for the frontend, concerning the backend, since it contains some sensitive information I have not made it public

Host a static website on Google Cloud Storage Functions Framework for Python Automated static website publishing with Cloud Build Invalidate cached content Managing infrastructure as code with Terraform, Cloud Build, and GitOps Export your Google Cloud resources into Terraform format