Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-649

Public Cloud - Scale Out - Scale Up/Down - Increase Storage (how-to cloud plugins usage)

    XMLWordPrintable

    Details

    • Type: Explanation
    • Status: Published
    • Affects Version/s: Exasol 6.1.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Symptoms:
      Hide

      This solution explains the usage of the exaoperation cloud plugins.

      Show
      This solution explains the usage of the exaoperation cloud plugins.
    • Explanation:
      Hide
      Plugin version 1.1 or greater is required to perform the actions described in this document. 

      The latest plugin version can be found in the download area: https://www.exasol.com/portal/display/DOWNLOAD/AWS+Plugins+and+AMIs

      Overview 

      This documentation explains how to easily perform the following operations using the cluster-integrated cloud plugin (Cloud-UI-Backend Plugin)

      • Scale-Out (Add nodes)*: Add one or more nodes to the database cluster and database
      • Reorganize Database:* Redistribute the data across the nodes after adding nodes
      • Scale-Up/Down (Change instance type): Change the instance type/model of all data nodes in order to increase or decrease the overall processing power
      • Increasing disk storage capacity: Increase the overall storage capacity by enlarging the data volumes of all database nodes.

      All operations can be launched via scripting or using the Cloud Plugin Web UI

      The cloud plugin creates and modifies the required resources on the respective cloud-platform and performs the required administration tasks in a fully automated manner. Typically, just a single command is needed to perform one the supported tasks.

       Please note that each of these operations causes short unavailability of the system  

       *Adding additional nodes to a cluster requires a database reorganization after the scale out operation that has to be seperately launched. As long as the REORGANIZE command is still running we recommend avoiding running queries which access large tables and therefore may try to create big indices. For more details please see SOL-263.

      UI-based Usage

      Public Cloud UI (public IP-address for management node required - not recommended for productive environments)

      In case a public IP-address is assigned to the management node, you can use the public Cloud-UI. After having specified your public-IP or DNS-name of the management node and your Exaoperation credentials of the admin user, you can apply your changes with just a few clicks. 

      Local Cloud UI

      Starting with Cloud UI plugin Version 1.1 the Web UI is part of the plugin and is accessible after plugin installation (for installation/upgrading instructions see SOL-640) by default on the management node on port 8835

       The Reorganize database action executes the  "REORGANIZE" statement.  In case of large databases the data reorganization may take a while. In order to speed up this process we recommend using multiple REORGANIZE streams as described in https://www.exasol.com/support/browse/SOL-263

      Script-based Usage

      Prerequisites

      •  The scripts only properly work if there is only one configured database in the cluster. For multi-database cluster setups this approach should not be used.
      • EXAoperation (including the failover-plugin) need direct access to the relevant AWS services (i.e. not through a proxy)
      • The plugin is preinstalled on AWS AMI >= 6.1.0-8
      • To verify the correct installation and version please check in EXAoperation if the plugin (Cloud.UIBackend-1.0.0)  appears in the software section. The sources should exist in /opt/ui-backend/* on the management node.

         

      • Please check for an updated version of the plugin and install/update the plugin according to SOL-640 if required/recommended.
      • in case of AWS, the EC2 Service has to be reachable and the management node needs the correct IAM permission (permissions should be properly configured if the cluster was deployed using the Exasol cloudformation templates from SOL-605 or the cloud installation wizard)  

      How to execute the cloud plugin operations

      The plugin can either be called locally directly on the management node or remotely via the xmlrpc interface

      Running locally on the management node:

      # Example
      
      # Adding 1 node with 16 GB physical RAM to the cluster and the database.
      # DB is enlarged but not reorganized
      # The parameter "memory" specifies the pyhsical memory per node. The overall amount of main memory used by the database is automatically adjusted. 
      # In the example below SYS_PASSWORD is a placeholder for the password of the database sys user;  EXAOPERATION_PASSWORD is a placeholder for the admin user that is used for the administration interface ExaOperation 
      
      /opt/ui-backend-plugin/handle_cloudui_request.py -d '{"method":"add_nodes","number":"1","memory":"16","credentials":"'$(echo -n 'admin:EXAOPERATION_PASSWORD' | base64)'"}'
      
      # Please note that after nodes have been added to the cluster a "REORGANIZE DATABASE" has to be executed to fully integrate these nodes into the database. This statement redistributes the data across the nodes and reconstitutes the distribution status.
      
      
      # configure the size of all data disks (of all db nodes) to 500 GB (this value must be bigger than the existing 
      data disk size; keep in mind to size the correct overall storage [disk x number_of_disk_per_db_node x db_nodes])
      
      /opt/ui-backend-plugin/handle_cloudui_request.py -d '{"method":"set_new_data_disk_size","value":500,"credentials":"'$(echo -n 'admin:EXAOPERATION_PASSWORD' | base64)'"}'
      
      
      
      # change the instance type of all db nodes to the new type m5.2xlarge, the physical RAM size of the new instance type has to be specified, as the overall amount of main memory used by the database (DBRam) is automatically adjusted. 
      
      /opt/ui-backend-plugin/handle_cloudui_request.py -d '{"method":"change_instance_type","value":"m5.2xlarge","memory":"32","credentials":"'$(echo -n 'admin:EXAOPERATION_PASSWORD' | base64)'"}'
      
      

       Please note that after nodes have been added to the cluster a "REORGANIZE DATABASE" has to be executed to fully integrate these nodes into the database. This statement redistributes the data across the nodes and reconstitutes the distribution status.

      You can initiate the reorganization via the following command

      /opt/ui-backend-plugin/handle_cloudui_request.py -d '{"method":"reorganize_database","db_password":"SYS_PASSWORD","reorganize_method":"reorganize","credentials":"'$(echo -n 'admin:EXAOPERATION_PASSWORD' | base64)'"}'

      This command  executes the  "REORGANIZE DATABASE" statement.  In case of large databases the data reorganization may take a while. In order to speed up this process we recommend using multiple REORGANIZE streams as described in https://www.exasol.com/support/browse/SOL-263

      Running remotely from a different host

       If you don't have public access to the management node

      In case of not having public access, you can use the attached python command line tools [^Cloud-UI-Backend-scripts.zip]. Those scripts can be executed for example from a jump host or a bastion server that resides in the same private network as the management node of the cluster. 

      # Example calls
      
      python add_additional_nodes.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --memory '8' --number '1'
      
      # Please note that after nodes have been added to the cluster a "REORGANIZE DATABASE" has to be executed to fully integrate these nodes into the database. This statement redistributes the data across the nodes and reconstitutes the distribution status.
      
      python change_data_disks_size.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --new_disk_size '500'
      
      python change_instance_types.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --instance_type 'm5.large' --memory '8'
      
      

      Running the reorganize command remotely from a different host (e.g. with the provided [^Cloud-UI-Backend-scripts.zip]) :

      python reorganize_database.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --passworddb 'password_for_user_sys' --reorganize_method 'reorganize' 

      This command  executes the  "REORGANIZE DATABASE" statement.  In case of large databases the data reorganization may take a while. In order to speed up this process we recommend using multiple REORGANIZE streams as described in https://www.exasol.com/support/browse/SOL-263

      Show
      Plugin version 1.1 or greater is required to perform the actions described in this document.  The latest plugin version can be found in the download area: https://www.exasol.com/portal/display/DOWNLOAD/AWS+Plugins+and+AMIs Overview  This documentation explains how to easily perform the following operations using the cluster-integrated cloud plugin (Cloud-UI-Backend Plugin) Scale-Out (Add nodes) *: Add one or more nodes to the database cluster and database Reorganize Database :* Redistribute the data across the nodes after adding nodes Scale-Up/Down (Change instance type): Change the instance type/model of all data nodes in order to increase or decrease the overall processing power Increasing disk storage capacity : Increase the overall storage capacity by enlarging the data volumes of all database nodes. All operations can be launched via scripting or using the Cloud Plugin Web UI The cloud plugin creates and modifies the required resources on the respective cloud-platform and performs the required administration tasks in a fully automated manner. Typically, just a single command is needed to perform one the supported tasks.  Please note that each of these operations causes short unavailability of the system     *Adding additional nodes to a cluster requires a database reorganization after the scale out operation that has to be seperately launched. As long as the REORGANIZE command is still running we recommend avoiding running queries which access large tables and therefore may try to create big indices. For more details please see SOL-263. UI-based Usage Public Cloud UI (public IP-address for management node required - not recommended for productive environments ) In case a public IP-address is assigned to the management node, you can use the public  Cloud-UI . After having specified your public-IP or DNS-name of the management node and your Exaoperation credentials of the admin user, you can apply your changes with just a few clicks.  Local Cloud UI Starting with Cloud UI plugin Version 1.1 the Web UI is part of the plugin and is accessible after plugin installation (for installation/upgrading instructions see  SOL-640 ) by default on the management node on port 8835  The Reorganize database action executes the  "REORGANIZE" statement.  In case of large databases the data reorganization may take a while. In order to speed up this process we recommend using multiple REORGANIZE streams as described in https://www.exasol.com/support/browse/SOL-263 Script-based Usage Prerequisites  The scripts only properly work if there is only one configured database in the cluster. For multi-database cluster setups this approach should not be used. EXAoperation (including the failover-plugin) need direct access to the relevant AWS services (i.e. not through a proxy) The plugin is preinstalled on AWS AMI >= 6.1.0-8 To verify the correct installation and version please check in EXAoperation if the plugin ( Cloud.UIBackend-1.0.0 )  appears in the software section. The sources should exist in /opt/ui-backend/ * on the management node.     Please check for an updated version of the plugin and install/update the plugin according to  SOL-640  if required/recommended. in case of AWS, the EC2 Service has to be reachable and the management node needs the correct IAM permission (permissions should be properly configured if the cluster was deployed using the Exasol cloudformation templates from SOL-605  or the cloud installation wizard)   How to execute the cloud plugin operations The plugin can either be called locally directly on the management node or remotely  via the xmlrpc interface Running locally on the management node : # Example # Adding 1 node with 16 GB physical RAM to the cluster and the database. # DB is enlarged but not reorganized # The parameter "memory" specifies the pyhsical memory per node. The overall amount of main memory used by the database is automatically adjusted. # In the example below SYS_PASSWORD is a placeholder for the password of the database sys user; EXAOPERATION_PASSWORD is a placeholder for the admin user that is used for the administration interface ExaOperation /opt/ui-backend-plugin/handle_cloudui_request.py -d '{ "method" : "add_nodes" , "number" : "1" , "memory" : "16" , "credentials" : "' $(echo -n 'admin:EXAOPERATION_PASSWORD' | base64) '" }' # Please note that after nodes have been added to the cluster a "REORGANIZE DATABASE" has to be executed to fully integrate these nodes into the database. This statement redistributes the data across the nodes and reconstitutes the distribution status. # configure the size of all data disks (of all db nodes) to 500 GB ( this value must be bigger than the existing data disk size; keep in mind to size the correct overall storage [disk x number_of_disk_per_db_node x db_nodes]) /opt/ui-backend-plugin/handle_cloudui_request.py -d '{ "method" : "set_new_data_disk_size" , "value" :500, "credentials" : "' $(echo -n 'admin:EXAOPERATION_PASSWORD' | base64) '" }' # change the instance type of all db nodes to the new type m5.2xlarge, the physical RAM size of the new instance type has to be specified, as the overall amount of main memory used by the database (DBRam) is automatically adjusted. /opt/ui-backend-plugin/handle_cloudui_request.py -d '{ "method" : "change_instance_type" , "value" : "m5.2xlarge" , "memory" : "32" , "credentials" : "' $(echo -n 'admin:EXAOPERATION_PASSWORD' | base64) '" }'  Please note that after nodes have been added to the cluster a "REORGANIZE DATABASE" has to be executed to fully integrate these nodes into the database. This statement redistributes the data across the nodes and reconstitutes the distribution status. You can initiate the reorganization via the following command /opt/ui-backend-plugin/handle_cloudui_request.py -d '{ "method" : "reorganize_database" , "db_password" : "SYS_PASSWORD" , "reorganize_method" : "reorganize" , "credentials" : "' $(echo -n 'admin:EXAOPERATION_PASSWORD' | base64) '" }' This command  executes the  "REORGANIZE DATABASE" statement.  In case of large databases the data reorganization may take a while. In order to speed up this process we recommend using multiple REORGANIZE streams as described in https://www.exasol.com/support/browse/SOL-263 Running remotely from a different host :    If you don't have public access to the management node In case of not having public access, you can use the attached python command line tools [^Cloud-UI-Backend-scripts.zip] . Those scripts can be executed for example from a jump host or a bastion server that resides in the same private network as the management node of the cluster.  # Example calls python add_additional_nodes.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --memory '8' --number '1' # Please note that after nodes have been added to the cluster a "REORGANIZE DATABASE" has to be executed to fully integrate these nodes into the database. This statement redistributes the data across the nodes and reconstitutes the distribution status. python change_data_disks_size.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --new_disk_size '500' python change_instance_types.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --instance_type 'm5.large' --memory '8' Running the reorganize command remotely from a different host (e.g. with the provided  [^Cloud-UI-Backend-scripts.zip] ) : python reorganize_database.py --dns_management_node '192.168.0.10' --exaoperation_password 'exaoperation_password' --passworddb 'password_for_user_sys' --reorganize_method 'reorganize'   This command  executes the  "REORGANIZE DATABASE" statement.  In case of large databases the data reorganization may take a while. In order to speed up this process we recommend using multiple REORGANIZE streams as described in https://www.exasol.com/support/browse/SOL-263
    • Category 1:
      Plattform Support - EXASOL on AWS
    • Category 2:
      Cluster Administration - Cluster Management

      Attachments

        Issue Links

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: