Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-605

AWS Cluster Installation via CloudFormation Template

    XMLWordPrintable

    Details

    • Type: How To
    • Status: Obsolete
    • Affects Version/s: Exasol 6.1.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Solution:
      Hide

      The cluster deployment with ready-to-use CloudFormation templates is the easiest way to install an EXASOL Cluster.
      This Getting Started guide helps you get up and running with EXASOL quickly.

      If you have to install an exasol cluster with a version 6.0 for updating purposes from version 5.x, then have a look at SOL-654

      This solution covers the following steps:

      1. Prerequisites
      2. Selecting an appropriate instance type and storage
      3. Configuring the instance and launching the deployment
      4. Logging into EXAoperation
      5. Connecting using EXAplus

      1. Prerequisites:

      In order to deploy Exasol using CloudFormation you have to subscribe to the corresponding offering in the AWS Marketplace first.

      • Sign in into the aws marketplace and subscribe to the corresponding EXASOL offering under:
      • When your subscription process has been completed you get a notification by mail (alternatively you can verify if the AMIs are available by typing "exasol" in the search field for public EC2 AMIs in the AWS Console)

      2. Selecting appropriate instance types and storage size

      A good rule of thumb is that the total amount of RAM of the cluster should correspond to approximately 10% - 15 % of your raw data.

      The template is preconfigured for approximately 2.5-3TB of raw data (5 x m4.4xlarge with 16 vcores, 64 GB; 320GB Memory in total). If this configuration is fine for you, you might skip the rest of this section.

      EXASOL supports all EC2 instances with >15GB Main Memory.
      We recommend that you use instance types of category r4,r5,m4,m4 or c5 with >30 GB main memory.

      If we assume that your raw data (uncompressed) is about 2.5TB in size, Therefore following configuration could be a good choice:

      5 * m4.4xlarge(16 vcores, 64 GB RAM) = 320 GB main memory in total -> This main memory size corresponds to 10%-15% of the raw data size.

      Please note:

      • The sizing "rule of thumb" used above may not be appropriate for your use case. Based on your specific use case, you may need significantly more memory.
      • For more information on sizing please refer to: EXASOL Sizing and the corresponding section in the FAQ document on installation and configuration

      The data storage volumes should be properly sized in order to store the maximum recommended amount of data for this instance type, data replicas in case of a cluster configuration as well as index structures and space for temporary data.

      This template already configures the optimal storage configuration for every instance type (using data redundancy 2: every data block is stored 2 times)

      When selecting "r3.8xlarge-2.928TB--EBS" as "Instance Type and Storage", the optimal number of EBS drives with a total size of 2.928TB will be automatically attached to every node of type r3.8xlarge.

      If you would like to configure the storage independently from the instance types please use the templates in SOL-636

      3. Configuring the cluster and launching the deployment

      Depending on your use case please select one of the available templates:

      • Use the left button to deploy the database sytem into an existing VPC infrastructure (just the target subnet needs to be specified):
      • Use the right button (generate new VPC) if you want to also deploy new network infrastructure (including VPC, PublicSubnet, InternetGateway, RouteTables, Security Group)

      (use existing VPC) (generate new VPC)

      This list contains all parameters for both templates:

      Parameter Usage/Description
      Stack name Name of the AWS CloudFormation Stack
      Database Name Name of the generated database
      SYS user password Password for the exasol database administration user (SYS)
      ADMIN user Password Password for the EXAoperation system administration user (ADMIN)
      VPC CIDR Block If a new VPC is created, a valid CIDR Block has to be specified
      Database Subnet-Id (only when deploying into existing VPC) Choose a SubnetID from the dropdown menu
      Subnet CIDR Block Choose a valid CIDR Block if deplyoment into new VPC (Optional)
      Database Placement Group An existing placement group can be choosen here, else a new one is created
      Public IPs If true, associate public ip addresses to all instances
      DNS Server DNS Server (default AWS DNS Server 169.254.169.253)
      System Timezone Required timezone for the DB (default Europe/Berlin)
      License Server IP IP address of the License Server (Data Nodes IP are counting upwards from this one) (Optional)
      License Server Instance Type EC2 instance type of the license server
      AMI ID Select corresponding AMI from the AWS Marketplace. The default setting is the latest PAYG image.
      First Data Node IP Counting upwards for the IPs of the data nodes (FirstDataNodeIP > LicenceServerIP) (Optional)
      Number of Data Nodes Number of database nodes that store data and process queries (min:2, max:64)
      Data Node Instance Type and Storage Instance type and EBS storage configuration of data nodes
      Replication Factor Replication Factor defines how many copies of a data block are kept in the cluster (1 means: no redundancy)
      StandByNodes If the replication factor is > 1 then a standby node can automatically replace a failed node
      Encrypt EBS Volumes Enable encryption of block storage
      Remote Access From IP 0.0.0.0/0 if allow access from anywhere
      AWS Key Pair Choose keypair for SSH access of created instances
      License In case of a BYOL image, an already aquired license can be pasted here
      • Click next to proceed to the options view (normally no further configuration is needed here)
      • After reviewing your configuration parameters confirm the message I acknowledge that AWS CloudFormation might create IAM resources. (Remark: This message is displayed because the template creates an AWS-Lambda resource that needs IAM permissions to launch the EC2 instances)
        Now it will take some time (10-25 minutes) until the database node is available.

      Determinig the public IP of the database nodes

      Navigate to  the EC2-Management-Console and identify the EC2 instances that have been created.

      • The name for the License-Server is [Stack-Name]-management-node
      • The names for the database nodes are [Stack-Name]-data-node

      When selecting one of the database nodes the corresponding public IP address is displayed after the label "Public DNS (IPv4)"

      4. Logging into the administration frontend EXAoperation

      Please note: It may take a while until the web based administration application EXAopertion becomes available after deployment (this may take up to 45 minutes)

      When the installation has been finialized, you can access the EXAoperation Web-Interface at https://<public_dns>/ (For determing the public IP address please refer to the last section.).
      Please confirm the digital certificate first as its validity cannot be automatically verified by the web browser ("Your connection is not secure").

      You can log into EXAoperation using the username 'admin' and the password defined for this user.
      If you can open the login page but you cannot login using the password you defined during the deployment, the installation might be still in progress. Access to EXAoperation will not be unlocked unless the installation has been finished.

      Please copy (via Strg-C) the connection string (e.g. 123.12.34.56.11...15:8563) that is displayed on the EXAoperation overview page.

      5. Connecting using a SQL Client

      You need to first download and install a compatible SQL Client (e.g. DBVisualizer, DataGrip, DBeaver or EXAplus).
      You can download EXAplus from Exasol Download Section.

      If you defined Public IPs: true during deployment, port 8563, Exasol's standard communication port can be accessed from outside AWS. This can be changed in the security group configuration.

      Open EXAplus and log in with your Sys username and password, which you created in the previous step and the connection string you retrieved from EXAoperation.

       

      Show
      The cluster deployment with ready-to-use CloudFormation templates is the easiest way to install an EXASOL Cluster. This Getting Started guide helps you get up and running with EXASOL quickly. If you have to install an exasol cluster with a version 6.0 for updating purposes from version 5.x, then have a look at  SOL-654 .  This solution covers the following steps: Prerequisites Selecting an appropriate instance type and storage Configuring the instance and launching the deployment Logging into EXAoperation Connecting using EXAplus 1. Prerequisites: In order to deploy Exasol using CloudFormation you have to subscribe to the corresponding offering in the AWS Marketplace first. Sign in into the aws marketplace and subscribe to the corresponding EXASOL offering under: the Pay-As-You-Go (PAYG) offering:  https://aws.amazon.com/marketplace/pp/B07M718TBR the Bring-Your-Own-License (BYOL): offering: https://aws.amazon.com/marketplace/pp/B01IVOPPN8 When your subscription process has been completed you get a notification by mail (alternatively you can verify if the AMIs are available by typing "exasol" in the search field for public EC2 AMIs in the AWS Console) 2. Selecting appropriate instance types and storage size A good rule of thumb is that the total amount of RAM of the cluster should correspond to approximately 10% - 15 % of your raw data. The template is preconfigured for approximately 2.5-3TB of raw data (5 x m4.4xlarge with 16 vcores, 64 GB; 320GB Memory in total). If this configuration is fine for you, you might skip the rest of this section. EXASOL supports all EC2 instances with >15GB Main Memory. We recommend that you use instance types of category r4,r5,m4,m4 or c5 with >30 GB main memory. If we assume that your raw data (uncompressed) is about 2.5TB in size, Therefore following configuration could be a good choice: 5 * m4.4xlarge(16 vcores, 64 GB RAM) = 320 GB main memory in total -> This main memory size corresponds to 10%-15% of the raw data size. Please note: The sizing "rule of thumb" used above may not be appropriate for your use case. Based on your specific use case, you may need significantly more memory. For more information on sizing please refer to: EXASOL Sizing and the corresponding section in the FAQ document on installation and configuration The data storage volumes should be properly sized in order to store the maximum recommended amount of data for this instance type, data replicas in case of a cluster configuration as well as index structures and space for temporary data. This template already configures the optimal storage configuration for every instance type (using data redundancy 2: every data block is stored 2 times) When selecting "r3.8xlarge-2.928TB--EBS" as "Instance Type and Storage", the optimal number of EBS drives with a total size of 2.928TB will be automatically attached to every node of type r3.8xlarge. If you would like to configure the storage independently from the instance types please use the templates in SOL-636 3. Configuring the cluster and launching the deployment Depending on your use case please select one of the available templates: Use the left button to deploy the database sytem into an existing VPC infrastructure (just the target subnet needs to be specified): Use the right button ( generate new VPC ) if you want to also deploy new network infrastructure (including VPC, PublicSubnet, InternetGateway, RouteTables, Security Group) (use existing VPC) (generate new VPC) This list contains all parameters for both templates: Parameter Usage/Description Stack name Name of the AWS CloudFormation Stack Database Name Name of the generated database SYS user password Password for the exasol database administration user (SYS) ADMIN user Password Password for the EXAoperation system administration user (ADMIN) VPC CIDR Block If a new VPC is created, a valid CIDR Block has to be specified Database Subnet-Id (only when deploying into existing VPC) Choose a SubnetID from the dropdown menu Subnet CIDR Block Choose a valid CIDR Block if deplyoment into new VPC (Optional) Database Placement Group An existing placement group can be choosen here, else a new one is created Public IPs If true, associate public ip addresses to all instances DNS Server DNS Server (default AWS DNS Server 169.254.169.253) System Timezone Required timezone for the DB (default Europe/Berlin) License Server IP IP address of the License Server (Data Nodes IP are counting upwards from this one) (Optional) License Server Instance Type EC2 instance type of the license server AMI ID Select corresponding AMI from the AWS Marketplace. The default setting is the latest PAYG image. First Data Node IP Counting upwards for the IPs of the data nodes (FirstDataNodeIP > LicenceServerIP) (Optional) Number of Data Nodes Number of database nodes that store data and process queries (min:2, max:64) Data Node Instance Type and Storage Instance type and EBS storage configuration of data nodes Replication Factor Replication Factor defines how many copies of a data block are kept in the cluster (1 means: no redundancy) StandByNodes If the replication factor is > 1 then a standby node can automatically replace a failed node Encrypt EBS Volumes Enable encryption of block storage Remote Access From IP 0.0.0.0/0 if allow access from anywhere AWS Key Pair Choose keypair for SSH access of created instances License In case of a BYOL image, an already aquired license can be pasted here Click next to proceed to the options view (normally no further configuration is needed here) After reviewing your configuration parameters confirm the message I acknowledge that AWS CloudFormation might create IAM resources. (Remark: This message is displayed because the template creates an AWS-Lambda resource that needs IAM permissions to launch the EC2 instances) Now it will take some time (10-25 minutes) until the database node is available. Determinig the public IP of the database nodes Navigate to  the EC2-Management-Console and identify the EC2 instances that have been created. The name for the License-Server is  [Stack-Name] -management-node The names for the database nodes are [Stack-Name] -data-node When selecting one of the database nodes the corresponding public IP address is displayed after the label "Public DNS (IPv4)" 4. Logging into the administration frontend EXAoperation Please note: It may take a while until the web based administration application EXAopertion becomes available after deployment (this may take up to 45 minutes) When the installation has been finialized, you can access the EXAoperation Web-Interface at https://<public_dns>/ (For determing the public IP address please refer to the last section.). Please confirm the digital certificate first as its validity cannot be automatically verified by the web browser ("Your connection is not secure"). You can log into EXAoperation using the username 'admin' and the password defined for this user. If you can open the login page but you cannot login using the password you defined during the deployment, the installation might be still in progress. Access to EXAoperation will not be unlocked unless the installation has been finished. Please copy (via Strg-C) the connection string (e.g. 123.12.34.56.11...15:8563 ) that is displayed on the EXAoperation overview page. 5. Connecting using a SQL Client You need to first download and install a compatible SQL Client (e.g. DBVisualizer, DataGrip, DBeaver or EXAplus). You can download EXAplus from Exasol Download Section . If you defined Public IPs: true during deployment, port 8563, Exasol's standard communication port can be accessed from outside AWS. This can be changed in the security group configuration. Open EXAplus and log in with your Sys username and password, which you created in the previous step and the connection string you retrieved from EXAoperation.  
    • Category 1:
      Plattform Support - EXASOL on AWS
    • Category 2:
      Cluster Administration - Installation

      Attachments

        Issue Links

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: