Deploy a Semarchy xDM instance in AWS

You will be guided to deploy Semarchy xDM on AWS. The estimated time to complete these steps is two hours.

It is highly recommended not to use the root user for any deployment or operations described below. You are advised to always follow the policy of least privilege for all access granted as part of the deployment, such as Security Groups.

Step 1. Check the AWS Credentials

Before starting the deployment check that the AWS Command Line Interface (CLI) is correctly configured:

  1. Run the following command to retrieve your AWS IAM user:

    $ aws iam get-user
    Example 1. Command Output
    {
        "User": {
            "Path": "/",
            "UserName": "<username>",
            "UserId": "<user_id>",
            "Arn": "arn:aws:iam::<account_id>:user/<username>",
            "CreateDate": "2022-02-08T16:19:26+00:00",
            "PasswordLastUsed": "2022-04-21T08:41:18+00:00"
        }
    }

    If the output is different it means that AWS CLI is not configured correctly. Refer to Getting started with the AWS CLI for more information.

  2. All the following commands will be executed on the default AWS region. Run the following command to confirm the default region used by AWS CLI:

    $ aws ec2 describe-availability-zones
        --output text
        --query 'AvailabilityZones[0].[RegionName]'
    Example 2. Command Output
    eu-west-1

    Refer to Configuration basics if the region is not set correctly.

Step 2. (Optional) Create the Aurora RDS instance

This step is optional. If you already have a running database instance, you can proceed to the next section.

This guide provides AWS CLI commands to create the database instance. This can also be achieved using the AWS console UI. Refer to interface reference for Amazon RDS for more information on AWS CLI for RDS.

These commands only provide mandatory parameters but you need to ensure they meet your requirements. Refer to AWS CLI Command Interface for more information.

Start by creating the database instance into which the Semarchy xDM repository and, optionally, the data locations for your applications, are stored.

  1. Run the following command to create the database cluster:

    $ aws rds create-db-cluster
        --db-cluster-identifier <db_cluster_name>
        --engine aurora-postgresql
        --master-username <db_clsuter_username>
        --master-user-password <db_cluster_password>
        --database-name semarchy_repository
        --tags Key=xdm-eks
  2. Run the following command to create the writer instance:

    $ aws rds create-db-instance
        --db-cluster-identifier <db_cluster_name>
        --db-instance-identifier <writer_instance_name>
        --db-instance-class db.t4g.medium
        --engine aurora-postgresql
        --db-subnet-group-name <db_subnet_group_name>
        --vpc-security-group-ids <security_group_id>
        --tags Key=xdm-eks
    Make sure that the DB subnet group and the security group are part of the same VPC.
  3. Optionally, add reader instances to dispatch the load between multiple reader instance(s) for read-only operations such as dashboards and BI tools:

    $ aws rds create-db-instance
        --db-cluster-identifier <db_cluster_name>
        --db-instance-identifier <reader_instance_name>
        --db-instance-class db.t4g.medium
        --engine aurora-postgresql
        --db-subnet-group-name <db_subnet_group_name>
        --vpc-security-group-ids <security_group_id>
        --tags Key=xdm-eks
  4. Follow the cluster and instance creation progress using the AWS console and wait for their status to change to Available (estimated time: 10 minutes). Note the endpoint URLs for later:

    AWS RDBS Console

    By default, the Aurora database cluster is not encrypted. Refer to Encrypting Amazon Aurora resources for more information.

Step 3. (Optional) Create the EKS Cluster

This step is optional. If you already have an EKS cluster configured with your AWS account, you can move to the next section.

This section guides you to create the EKS cluster needed to deploy the Semarchy xDM images.

Use the eksctl tool to generate a series of AWS CLI command lines based on a configuration file. The tool will generate and execute two CloudFormation stacks resulting in the creation of the EKS cluster and a group node with three nodes.

  1. Download the EKS cluster configuration file template and edit the content of the file to match your specific requirements. Refer to Config file schema for more information.

  2. Run the following command from the folder containing the configuration file:

    $ eksctl create cluster --config-file <cluster_configuration_file>.yaml

    The operation takes approximately 10 to 20 minutes.

  3. Open the AWS EKS console and check whether the cluster and group node are created and have an Active and Ready status respectively:

    AWS EKS Console

  4. Run the following command to configure kubectl with your cluster. Refer to Create a kubeconfig for Amazon EKS for more information.

    $ aws eks update-kubeconfig
        --region <region_id>
        --name <cluster_name>
    Example 3. Command Output
    Added new context
    arn:aws:eks:<region_id>:<aws_accout_id>:cluster/<cluster_name> to
    /Users/<username>/.kube/config
  5. Run the following command to test your kubectl configuration:

    $ kubectl get svc
    Example 4. Command Output
    NAME        TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)  AGE
    kubernetes  ClusterIP  172.20.0.1  <none>       443/TCP  23m

Step 4. (Optional) Create the Ingress Load Balancer Cluster

This step is optional. If you already have a Load Balancer configured with your EKS cluster, you can proceed to the next section.

The Load Balancer is necessary to route users to the active instance or the passive instances. In the example, you will be using the NGINX Ingress Controller but it can be replaced by any Load Balancer supporting sticky sessions (it is mandatory for the passive instances).

  1. Run the following command to install the NGINX Ingress Controller:

    $ kubectl apply -f
    https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.2/deploy/static/provider/cloud/deploy.yaml
    Example 5. Command Output
    namespace/ingress-nginx created
    serviceaccount/ingress-nginx created
    serviceaccount/ingress-nginx-admission created
    role.rbac.authorization.k8s.io/ingress-nginx created
    role.rbac.authorization.k8s.io/ingress-nginx-admission created
    clusterrole.rbac.authorization.k8s.io/ingress-nginx created
    clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
    rolebinding.rbac.authorization.k8s.io/ingress-nginx created
    rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
    clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
    clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
    configmap/ingress-nginx-controller created
    service/ingress-nginx-controller created
    service/ingress-nginx-controller-admission created
    deployment.apps/ingress-nginx-controller created
    job.batch/ingress-nginx-admission-create created
    job.batch/ingress-nginx-admission-patch created
    ingressclass.networking.k8s.io/nginx created
    validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
  2. Run the following command to retrieve the internal IP of the Load Balancer:

    $ kubectl get svc -n ingress-nginx ingress-nginx-controller
    Example 6. Command Output
    NAME                      TYPE          CLUSTER-IP    EXTERNAL-IP    PORT(S)                    AGE
    ingress-nginx-controller  LoadBalancer  <Cluster_IP>  <External_IP>  80:31347/TCP,443:32449/TCP 29s
    Take note of the cluster Cluster IP; it will be used later to confirm that the deployment was successful.

Step 5. Update Security Group

At this step, you need to authorize the EKS cluster to connect to the Aurora cluster before completing the deployment.

Edit the Inbound Rules of the Security Group of your aurora writer instance:

  1. Navigate to AWS console > RDS.

  2. From your Aurora cluster, select the writer instance and navigate to Connectivity & Security > Security, and edit the VPC Security Group.

  3. Navigate to the Inbound Rules tab and select Edit Inbound rules.

  4. Select Add rule and set the following values:

    • Type: PostgreSQL

    • Source: eks-cluster-sg-xdm-eks-<ressource_name>

      Use the search tool to find the source.
    • Name: EKS cluster

  5. Click Save to validate.

Step 6. Set the ConfigMap

At this step, you need to define a Kubernetes ConfigMap to set environment variables shared with every pod you deploy.

All the following Kubernetes commands are executed in the default namespace.

Seven environment variables are defined and used for the Semarchy xDM startup configuration:

  • SEMARCHY_SETUP_TOKEN: The setup token that you need to enter during the Semarchy Repository creation.

  • XDM_REPOSITORY_DRIVER: JDBC driver class for the repository database. Leave the default value for a Postgres database.

  • XDM_REPOSITORY_URL: JDBC URL for the repository database. It needs to match your Database writer instance endpoint.

  • XDM_REPOSITORY_USERNAME: Database user to connect the repository database.

  • XDM_REPOSITORY_PASSWORD: Database user password.

  • XDM_REPOSITORY_READONLY_USERNAME: Database read-only user to connect the repository database.

  • XDM_REPOSITORY_READONLY_PASSWORD: Database read-only user password.

It is mandatory to modify the above values (except XDM_REPOSITORY_DRIVER).
  1. Download the sample manifest file for the ConfigMap.

  2. Edit the file and save your modifications.

  3. Run the following command from the folder containing your manifest file:

    $ kubectl apply -f <configmap_file>.yaml
    Example 7. Command Output
    configmap/semarchy-config created

Step 7. Execute the SQL init script

At this step, you need to configure the database schemas required to create the Semarchy repository. As the Aurora cluster is not accessible on the internet by default, use the Kubernetes pod to access it and run the SQL initialization script. If necessary you can configure the Aurora cluster to make it available outside of the AWS VPC.

  1. Download the sample manifest file for the disposable pod (based on a Debian image). This pod will be on the same Virtual Network as the database instance(s) and will be able to access it.

  2. Run the following command to deploy a disposable pod:

    $ kubectl apply -f <disposable_pod_file>.yaml
    Example 8. Command Output
    pod/semarchy-disposable-pod created
  3. Run the following command until the pod is started (Status: Running):

    $ kubectl get pod semarchy-disposable-pod
    Example 9. Command Output
    NAME                     READY  STATUS   RESTARTS  AGE
    semarchy-disposable-pod  1/1    Running  0         16s

    The pod can take about ten to twenty seconds to get running.

  4. Download the SQL script and edit it to match the values you have set in the ConfigMap.

    • semarchy_repository: Database used for the repository.

    • semarchy_repository_username: Database username to connect to the repository database.

    • semarchy_repository_password: Database user password.

    • semarchy_repository_ro_username: Database read-only user to connect to the repository database.

    • semarchy_repository_ro_password: Database read-only user password.

  5. Save the file and run the following command to copy the script file to the disposable pod tmp folder:

    $ kubectl cp init-db.sql semarchy-disposable-pod:/tmp
  6. Run the following command to access the disposable pod bash:

    $ kubectl exec -it semarchy-disposable-pod -- bash
  7. Install the following curl command that you will use later:

    $ apk add curl
  8. Run the following command to go to the tmp folder:

    $ cd /tmp
  9. Run the following command to connect to the Database writer instance and to execute the initialization script:

    $ psql
        --host "<writer_instace_endpoint>"
        --username "<db_clsuter_username>"
        --dbname "semarchy_repository" < init-db.sql
    Example 10. Command Output
    Password for user <db_clsuter_username>:
  10. Enter the database cluster master password (<db_clsuter_password>) and press Enter:

    Example 11. Command Output
    CREATE SCHEMA
    GRANT
    ALTER DEFAULT PRIVILEGES
    ALTER DATABASE
    CREATE EXTENSION
    CREATE EXTENSION
    CREATE ROLE
    GRANT ROLE
    CREATE SCHEMA
    CREATE ROLE
    GRANT
    ALTER ROLE
    GRANT
  11. Run the following command to connect to the database writer instance with the xDM Repository user (XDM_REPOSITORY_USERNAME) created with the initialization script:

    $ psql
        --host "<writer_instace_endpoints>"
        --username "<xdm_repository_username>"
        --dbname "semarchy_repository"
    Example 12. Command Output
    Password for user <xdm_repository_username>:
  12. Enter the database cluster master password (<xdm_repository_password>) and press Enter.

  13. Run the following command to list the existing database schemas:

    $ \l
    Example 13. Command Output
    semarchy_repository=> \l
    List of databases
    Name                | Owner    | Encoding | Collate     | Ctype       | Access privileges
    --------------------+----------+----------+-------------+-------------+----------------------
    postgres            | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
    xxxxxxxx            | xxxxxxxx | UTF8     | en_US.UTF-8 | en_US.UTF-8 | xxxxxx=CTc/xxxxxx
    semarchy_repository | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
    …
  14. Exit from the PSQL command with the following command:

    $ exit
  15. Exit from the disposable pod with the following command. You will re-use this pod later to confirm the deployment completion:

    $ exit

Step 8. Deploy the active pod

Once the database instance and EKS cluster are running, you can deploy the docker image of the application server.
You need to start by deploying the active node with a unique pod:

  1. Download the sample manifest file for the application server active node.
    This file defines the deployment of the application server active node and the service associated with exposing the application. You can edit the content of the file to match your specific requirements. Refer to the Kubernetes documentation for more information.

  2. Run the following command from the folder containing the manifest file to deploy the active pod:

    $ kubectl apply -f <appserver_active_file>.yaml
    Example 14. Command Output
    deployment.apps/semarchy-appserver-active created
    service/semarchy-appserver-active created
  3. Run the following command to check the deployment progress until the status becomes Ready (this can take a few minutes):

    $ kubectl get deployments
    Example 15. Command Output
    NAME                       READY  UP-TO-DATE  AVAILABLE  AGE
    semarchy-appserver-active  1/1    1           1          13m

    The deployment can take about ten to twenty seconds to be ready.

Step 9. Deploy the passive pods

At this step, deploy two instances of the passive application server image for a high availability configuration.

  1. Download the sample manifest file for the application server passive node. This file defines the deployment of the application server passive node and the service associated to expose the app. You can edit the content of the file to match your specific requirements. Refer to the Kubernetes documentation for more information.

  2. Run the following command from the folder containing the manifest file:

    $ kubectl apply -f <appserver_passive_file>.yaml
    Example 16. Command Output
    deployment.apps/semarchy-appserver-passive created
    service/semarchy-appserver-passive created
  3. Execute the following command to ensure that the passive nodes are deployed and ready:

    $ kubectl get deployments
    Example 17. Command Output
    NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
    semarchy-appserver-active   1/1    1           1          13m
    semarchy-appserver-passive  2/2    2           2          12m

Step 10. Configure the Load Balancer

Finally, you need to expose your Kubernetes pods on your networks.

Set a load balancer using sticky sessions to route the users to the active and passive pods:

  1. Download the sample Ingress manifest file and edit it to match your requirements.
    This file deploys an Ingress resource and configures it to use the sticky sessions for the passive instances.
    Refer to the Kubernetes documentation for more information.

  2. Run the following command to apply the configuration:

    $ kubectl apply -f <ingress_file>.yaml
    Example 18. Command Output
    ingress.networking.k8s.io/ingress created

Step 11. Check the platform connection

At this step, you have deployed all the required resources to run Semarchy xDM on AWS with Kubernetes. You need the <cluster_ip> you have retrieved at Step 4.

By default, the pods are not exposed to the internet. Hence, you have to use the disposable pod to check the platform connection with the active and passive nodes:

  1. Run the following command to confirm that the load balancer is routing to the active instance:

    $ kubectl exec -it semarchy-disposable-pod -- curl -v --resolve
    semarchy-appserver-active:80:<cluster_ip>
    semarchy-appserver-active:80/semarchy/api/rest/probes/started
    Example 19. Command Output
    * Added semarchy-appserver-active:80:<cluster_ip> to DNS cache
    * Hostname semarchy-appserver-active was found in DNS cache
    * Trying <cluster_ip>:80...
    * Connected to semarchy-appserver-active (<cluster_ip>) port 80 (#0)
    > GET /semarchy/api/rest/probes/started HTTP/1.1
    > Host: semarchy-appserver-active
    > User-Agent: curl/7.80.0
    > Accept: /
    >
    * Mark bundle as not supporting multiuse
    < HTTP/1.1 204
    …
    This command executes a curl command on the disposable pod to query the REST API prob endpoints.
  2. Run the following command to confirm that the load balancer is routing to the passive instances:

    $ kubectl exec -it semarchy-disposable-pod -- curl -v --resolve
    semarchy-appserver-passive:80:<cluster_ip>
    semarchy-appserver-passive:80/semarchy/api/rest/probes/started
    Example 20. Command Output
    * Added semarchy-appserver-passive:80:<cluster_ip> to DNS cache
    * Hostname semarchy-appserver-passive was found in DNS cache
    * Trying <cluster_ip>:80...
    * Connected to semarchy-appserver-passive (<cluster_ip>) port 80 (#0)
    > GET /semarchy/api/rest/probes/started HTTP/1.1
    > Host: semarchy-appserver-passive
    > User-Agent: curl/7.80.0
    > Accept: /
    >
    * Mark bundle as not supporting multiuse
    < HTTP/1.1 204
    …
  3. Delete the disposable pod with the following command:

    $ kubectl delete pod semarchy-disposable-pod
    Example 21. Command Output
    pod "semarchy-disposable-pod" deleted