Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-403

AWS Installation Guide (Cluster)

    XMLWordPrintable

    Details

    • Type: How To
    • Status: Published
    • Affects Version/s: EXASolution 5.0.15
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Solution:
      Hide

      This how-to includes the following sections describing the deployment of a 2+1 cluster, we'll use the jump host (Amazon AMI) for running the installation script setup-cloud.sh and aws cli.

      1. Cluster installation with jump host for WAN access
      2. Cluster installation using AWS PUBLIC IPs for all cluster nodes

      If you need the database to be available through the Internet, set following database parameter to enforce protocol encryption for EXAplus clients and drivers otherwise traffic between client and database will be unencrypted.

      -forceProtocolEncryption=1
      

      Prerequisites

      1. AWS console access e.g. https://COMPANY.singin.aws.amazon.com/console
      2. EXASOL AMI-ID
      3. SSH Key Pair for the corresponding region
      4. setup-cloud.sh Shell Script for deploying the cluster (see SOL-408 on how to download and use setup-cloud.sh)
      5. AWS command line utilities (http://docs.aws.amazon.com/cli/latest/userguide/installing.html), utilities are pre-installed on the Amazon AMI
      6. File containing your AWS environment variables (KEY/SECRET/REGION) to access the ec2 API
        $ cat .settings_virginia
        export AWS_ACCESS_KEY=KEY-ID
        export AWS_SECRET_KEY=SECRET-KEY
        export EC2_URL=https://ec2.us-east-1.amazonaws.com
        

      1. Cluster installation with jump host for WAN access

      1. Create a new VPC (VPC with single public subnet) or use an existing one.
        In this how to we use:
      • 10.0.0.0/16 CIDR block
        AND
      • A 10.0.1.0/24 Public subnet within the CIDR block. The first subnet within the CIDR block 10.0.0.0/24 must not be used . See SOL-409 for more details.
        AND
      • S3 Endpoint for database backups (see https://www.exasol.com/support/browse/SOL-354 for configuring S3 remote backups)
        AND
      • S3 Endpoint for IMPORT/EXPORT

      Installation and configuration of the jump host (used for cluster nodes WAN access)

      1. Launch free tier eligible instance using the EC2 console (e.g. Amazon AMI t2.micro) used as an intermediate jump host and router
      2. Assign Public or Elastic IP
      3. Allow SSH incoming traffic
      4. Adjust instance storage if necessary
      5. Launch instance

      Configure jump host as router/gateway

      $ ssh -i PEM ec2-user@PUBLIC-IP
      jumphost $ sudo ec2-modify-instance-attribute --source-dest-check false <INSTANCE_ID_JUMP_HOST>
      jumphost $ sudo sysctl net.ipv4.ip_forward=1
      jumphost $ sudo iptables -t nat -A POSTROUTING -j MASQUERADE
      
      1. Adjust Security Group rules between jump host and cluster nodes (license node and data nodes) accordingly (All incoming traffic on both sides e.g. full subnet)
      2. Ensure DNS and NTP is configured and working properly

      Prepare jump host

      1. Copy setup-cloud.sh and .settings-file to jump host
      2. Source .settings and test ec2 API
        jumphost $ . settings 
        jumphost $ ec2-describe-regions 
        REGION  eu-west-1       ec2.eu-west-1.amazonaws.com
        REGION  ap-southeast-1  ec2.ap-southeast-1.amazonaws.com
        REGION  ap-southeast-2  ec2.ap-southeast-2.amazonaws.com
        REGION  eu-central-1    ec2.eu-central-1.amazonaws.com
        REGION  ap-northeast-2  ec2.ap-northeast-2.amazonaws.com
        REGION  ap-northeast-1  ec2.ap-northeast-1.amazonaws.com
        REGION  us-east-1       ec2.us-east-1.amazonaws.com
        REGION  sa-east-1       ec2.sa-east-1.amazonaws.com
        REGION  us-west-1       ec2.us-west-1.amazonaws.com
        REGION  us-west-2       ec2.us-west-2.amazonaws.com
        

      Create JSON.config file for setup-cloud.sh

      This example will setup a 2+1 cluster in Frankfurt using an r3.large instance as license node and three r3.xlarge instances as data nodes. All nodes are using 2x100GiB gp2 SSDs (first one for OS/SWAP/SPOOL and the second one for EXAStorage), EXAStorage is running in redundancy two and the database uses one standby node. The license node per default will get an AWS PUBLIC IP. See SOL-408 for full documentation on setup-cloud.sh and SOL-409 for subnet restrictions.

      {
          "VPC"                           :"vpc-id",
          "LICENSE_SERVER_INSTANCE_TYPE"  :"r3.large",   
          "DATANODE_INSTANCE_TYPE"        :"r3.xlarge",
          "AVAILABILITY_ZONE"             :"eu-central-1b",
          "KEYPAIR"                       :"JUMP_FRANKFURT_PEM",
          "IMAGE_ID"                      :"ami-id",
      
      
          "OS_EBS_SIZE"                   :"100",           
          "DATA_EBS_SIZE"                 :"100",          
          "EBS_DELETE_ON_TERMINATION"     :true,         
          "OS_EBS_TYPE"                   :"gp2",     
          "DATA_EBS_TYPE"                 :"gp2",
      
          "EXASOLUTION_PASSWORD"          :"your sys password",
          "EXAOPERATION_PASSWORD"         :"your admin password",
          
          "ST_REPLICATION_FACTOR"         :"2", 
      
          "DB_NAME"                       :"EXA1234_db1",
          "DB_PRIMARY_NODES"              :"2",         
          "DB_STANDBY_NODES"              :"1",      
          "DB_RESTORE_BACKUP_SOURCE"      :"",
      
          "DNS_SERVER_2"                  :"8.8.8.8",
          "TIMEZONE"                      :"Europe/Berlin",
          
          "NETWORK_RANGE"                 : "10.0.0.0/16",
          "DEFAULT_GATEWAY"               : "10.0.1.1",
      
          "LICENSE_SERVER_IP"             : "10.0.1.10"
      }
      

      Execute script

      jumphost$ ./setup-cloud.sh CONFIG.json 
      ...
      EC2 API output
      ... 
      

      The installation will take about 20 minutes to finish.

      SSH port forwarding on jump host for the database and web-frontend

      1. Ports that will be forwarded to your local machine (localhost):
        • 443 TCP HTTPS for EXAoperation web-frontend
        • 8563 TCP for connection to the EXASolution database
          $ ssh -i PEM ec2-user@JUMPHOST-IP -L8443:VPC-License-Node-IP:443 -L8563:VPC-Data-Node-IP:8563
          

          Alternatively, use the PUBLIC IP of the license node for port forwarding or connect directly through the Public IP of the license node (check if your Security Group Rules allow incoming traffic).

      Configure Search Domain

      1. EXAoperation - Network
        • Configure Search Domain to fit the region your cluster is running in e.g. Frankfurt otherwise S3 name lookup might fail e.g. "eu-central-1.compute.internal"

      2. Cluster installation using AWS PUBLIC IPs for all cluster nodes

      Use the same procedure as before, additionally edit setup-cloud.sh to launch data instances with PUBLIC IPs

      jumphost$ sed -i 's/--associate-public-ip-address False/--associate-public-ip-address True/' setup.cloud.sh
      
      Show
      This how-to includes the following sections describing the deployment of a 2+1 cluster, we'll use the jump host (Amazon AMI) for running the installation script setup-cloud.sh and aws cli. Cluster installation with jump host for WAN access Cluster installation using AWS PUBLIC IPs for all cluster nodes If you need the database to be available through the Internet, set following database parameter to enforce protocol encryption for EXAplus clients and drivers otherwise traffic between client and database will be unencrypted. -forceProtocolEncryption=1 Prerequisites AWS console access e.g. https://COMPANY.singin.aws.amazon.com/console EXASOL AMI-ID SSH Key Pair for the corresponding region setup-cloud.sh Shell Script for deploying the cluster (see SOL-408 on how to download and use setup-cloud.sh) AWS command line utilities ( http://docs.aws.amazon.com/cli/latest/userguide/installing.html ), utilities are pre-installed on the Amazon AMI File containing your AWS environment variables (KEY/SECRET/REGION) to access the ec2 API $ cat .settings_virginia export AWS_ACCESS_KEY=KEY-ID export AWS_SECRET_KEY=SECRET-KEY export EC2_URL=https://ec2.us-east-1.amazonaws.com 1. Cluster installation with jump host for WAN access Create a new VPC (VPC with single public subnet) or use an existing one. In this how to we use: 10.0.0.0/16 CIDR block AND A 10.0.1.0/24 Public subnet within the CIDR block. The first subnet within the CIDR block 10.0.0.0/24 must not be used . See SOL-409 for more details. AND S3 Endpoint for database backups (see https://www.exasol.com/support/browse/SOL-354 for configuring S3 remote backups) AND S3 Endpoint for IMPORT/EXPORT Installation and configuration of the jump host (used for cluster nodes WAN access) Launch free tier eligible instance using the EC2 console (e.g. Amazon AMI t2.micro) used as an intermediate jump host and router Assign Public or Elastic IP Allow SSH incoming traffic Adjust instance storage if necessary Launch instance Configure jump host as router/gateway $ ssh -i PEM ec2-user@PUBLIC-IP jumphost $ sudo ec2-modify-instance-attribute --source-dest-check false <INSTANCE_ID_JUMP_HOST> jumphost $ sudo sysctl net.ipv4.ip_forward=1 jumphost $ sudo iptables -t nat -A POSTROUTING -j MASQUERADE Adjust Security Group rules between jump host and cluster nodes (license node and data nodes) accordingly (All incoming traffic on both sides e.g. full subnet) Ensure DNS and NTP is configured and working properly Prepare jump host Copy setup-cloud.sh and .settings-file to jump host Source .settings and test ec2 API jumphost $ . settings jumphost $ ec2-describe-regions REGION eu-west-1 ec2.eu-west-1.amazonaws.com REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com REGION ap-southeast-2 ec2.ap-southeast-2.amazonaws.com REGION eu-central-1 ec2.eu-central-1.amazonaws.com REGION ap-northeast-2 ec2.ap-northeast-2.amazonaws.com REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com REGION us-east-1 ec2.us-east-1.amazonaws.com REGION sa-east-1 ec2.sa-east-1.amazonaws.com REGION us-west-1 ec2.us-west-1.amazonaws.com REGION us-west-2 ec2.us-west-2.amazonaws.com Create JSON.config file for setup-cloud.sh This example will setup a 2+1 cluster in Frankfurt using an r3.large instance as license node and three r3.xlarge instances as data nodes. All nodes are using 2x100GiB gp2 SSDs (first one for OS/SWAP/SPOOL and the second one for EXAStorage), EXAStorage is running in redundancy two and the database uses one standby node. The license node per default will get an AWS PUBLIC IP. See SOL-408 for full documentation on setup-cloud.sh and SOL-409 for subnet restrictions. { "VPC" :"vpc-id", "LICENSE_SERVER_INSTANCE_TYPE" :"r3.large", "DATANODE_INSTANCE_TYPE" :"r3.xlarge", "AVAILABILITY_ZONE" :"eu-central-1b", "KEYPAIR" :"JUMP_FRANKFURT_PEM", "IMAGE_ID" :"ami-id", "OS_EBS_SIZE" :"100", "DATA_EBS_SIZE" :"100", "EBS_DELETE_ON_TERMINATION" :true, "OS_EBS_TYPE" :"gp2", "DATA_EBS_TYPE" :"gp2", "EXASOLUTION_PASSWORD" :"your sys password", "EXAOPERATION_PASSWORD" :"your admin password", "ST_REPLICATION_FACTOR" :"2", "DB_NAME" :"EXA1234_db1", "DB_PRIMARY_NODES" :"2", "DB_STANDBY_NODES" :"1", "DB_RESTORE_BACKUP_SOURCE" :"", "DNS_SERVER_2" :"8.8.8.8", "TIMEZONE" :"Europe/Berlin", "NETWORK_RANGE" : "10.0.0.0/16", "DEFAULT_GATEWAY" : "10.0.1.1", "LICENSE_SERVER_IP" : "10.0.1.10" } Execute script jumphost$ ./setup-cloud.sh CONFIG.json ... EC2 API output ... The installation will take about 20 minutes to finish. SSH port forwarding on jump host for the database and web-frontend Ports that will be forwarded to your local machine (localhost): 443 TCP HTTPS for EXAoperation web-frontend 8563 TCP for connection to the EXASolution database $ ssh -i PEM ec2-user@JUMPHOST-IP -L8443:VPC-License-Node-IP:443 -L8563:VPC-Data-Node-IP:8563 Alternatively, use the PUBLIC IP of the license node for port forwarding or connect directly through the Public IP of the license node (check if your Security Group Rules allow incoming traffic). Configure Search Domain EXAoperation - Network Configure Search Domain to fit the region your cluster is running in e.g. Frankfurt otherwise S3 name lookup might fail e.g. "eu-central-1.compute.internal" 2. Cluster installation using AWS PUBLIC IPs for all cluster nodes Use the same procedure as before, additionally edit setup-cloud.sh to launch data instances with PUBLIC IPs jumphost$ sed -i 's/--associate-public-ip-address False/--associate-public-ip-address True/' setup.cloud.sh
    • Category 1:
      Plattform Support - EXASOL on AWS
    • Category 2:
      Cluster Administration - Installation

      Attachments

        Issue Links

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: