Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-261

Cluster Enlargement: Adding Additional Nodes

    Details

    • Type: How To
    • Status: Obsolete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: EXASolution, EXAStorage
    • Labels:
      None
    • Solution:
      Hide

      Synopsis

      This article guides through the enlargement of a cluster by way of example. It shows how to enlarge a 4-node system with data volume redundancy 2 by 2 additional nodes.

      Please note that

      1. The tasks performed in EXAoperation require a user with at least "Administrator" privileges.
      2. It is advisable to have a recent backup available offline (stored outside of the cluster).
      3. The "REORGANIZE DATABASE" command may be executed while other clients connect but it is recommended to execute it as exclusive task.
      4. We urge to either remove any database quota (Maximal Volume Size) during the enlargement/reorganize or to set it to a high enough value (see section 'Preparations' below).


      1. Preparations

      1.1. Review database instance quota and active nodes
      • Open 'Services > EXASolution'
      • Open the EXASolution instance by clicking on instance name

        Please note the number of active nodes for that database instance and the value in field Maximal Volume Size (GiB) and (an empty field indicates that there is no quota set).
      1.2. Review the volume size
      • Open 'Services > EXAStorage'
      • Note the size of the persistent volume.
      1.3. Calculations

      Database quotas limit the overall data volume (persistent & temporary tablespace) that may be allocated by an EXASolution instance. Redundancy is not taken into account.

      The quota must reflect the disk space requirements after the enlargement:

      1. The reorganize is a demanding operation and may trigger swapping out of blocks of data. We hence recommend to allow for some reserve (20% in this calculation).
      2. The quota must be bigger than the enlarged volume plus some overhead.
      1.2 * persistent volume size * number of nodes after enlargement / existing number of nodes 
      

      In our example:

      • volume size is 24GiB
      • existing number of nodes is 4
      • number of nodes after enlargement is 6
        1.2 * 24GiB * 6 / 4 = 44GiB

      2. Installing the nodes

      In Preparation for the installation of the new nodes you need to know the MAC addresses and which disk configuration is set on the running part of the cluster.

      2.1. Add node
      • Open 'Configuration > Nodes'
      • Click "Add"
        • configure the node to add
        • click apply


      • Repeat that for every node
      2.2. Install nodes
      • Open 'Configuration > Nodes'
      • Reload the page until the state of the recently added nodes changes to 'running'
      • Set the status of all installed nodes to "active"

      At the end is should look like this:

      3. Performing the enlargement

      3.1. Shutdown all databases
      • Open 'Services > EXASolution'
      • Check the database operations. If the database is stopped while an operation is in progress, the operation will be aborted
      • Select the EXASolution instances
      • Click "Shutdown"
      • Reload the page until all instances change their status from "Running" to "Created"
      • Again, you may follow the procedure in an appropriate logservice:

        System marked as stopped.
        Successfully send retry shutdown event to system partition 64.
        EXASolution exa_db is rejecting connections
        controller-st(0.0): Shutdown called.
        User 0 requests shutdown of system.

      3.2 Add nodes to storage
      • After the nodes have been installed, they'll be shown automatically in the storage overview
      • Add now the unused disks to the new nodes
      • The new nodes should look like a completely free node
      • Edit the volume by clicking on the name of the persistent volume
      • Click on edit
        Be sure, the enlarge will be done on the right volume
      • Add the two nodes and apply the configuration
      • The recently added nodes are now shown in the list
      3.3 Add nodes to Database
      • Open 'Services > EXASolution' and click on the database instance
      • Click on edit
      • Add the two nodes as reserve nodes to the database
      • Click apply
      3.4 Enlarge the database and start reorganize

      Check point 1.1 again. If the quota is exceeded, the database won't start again

      • Open the dropdown menu on the right side and submit "Enlarge"
      • Type in the number of master nodes and click apply
      • After applying, the database will start itself. Status is "Starting"
      • The status changes to "Running" if the database is up
      • Now login with EXAPlus-Client and start command :
         REORGANIZE DATABASE; 

      • Command has been started. For appraisal how long the Reorganize will take, feel free to contact the EXASOL support.
      • Reorganize has been done
      • Now the distribution of the disk usage should be approximately the same



      Additional information
      In case you want to enlarge with a number of nodes, which isn't a multiple of the redundancy you have chosen for the volume:
      You need to drop the redundancy before you can add the nodes to the volume. After you have added the nodes you can increase the redundancy again.
      As a result of dropping and increasing the redundancy, a recovery process will take place. DO NOT continue with the enlargement procedure during the recovery process
      e.g. It is possible to enlarge without dropping the redundancy, if you add two, four, six, eight, ... (a multiple of two) number of nodes to the cluster if the redundancy is two.

      Show
      Synopsis This article guides through the enlargement of a cluster by way of example. It shows how to enlarge a 4-node system with data volume redundancy 2 by 2 additional nodes. Please note that The tasks performed in EXAoperation require a user with at least "Administrator" privileges. It is advisable to have a recent backup available offline (stored outside of the cluster). The "REORGANIZE DATABASE" command may be executed while other clients connect but it is recommended to execute it as exclusive task. We urge to either remove any database quota (Maximal Volume Size) during the enlargement/reorganize or to set it to a high enough value (see section 'Preparations' below). 1. Preparations 1.1. Review database instance quota and active nodes Open 'Services > EXASolution' Open the EXASolution instance by clicking on instance name Please note the number of active nodes for that database instance and the value in field Maximal Volume Size (GiB) and (an empty field indicates that there is no quota set). 1.2. Review the volume size Open 'Services > EXAStorage' Note the size of the persistent volume. 1.3. Calculations Database quotas limit the overall data volume (persistent & temporary tablespace) that may be allocated by an EXASolution instance. Redundancy is not taken into account. The quota must reflect the disk space requirements after the enlargement: The reorganize is a demanding operation and may trigger swapping out of blocks of data. We hence recommend to allow for some reserve (20% in this calculation). The quota must be bigger than the enlarged volume plus some overhead. 1.2 * persistent volume size * number of nodes after enlargement / existing number of nodes In our example: volume size is 24GiB existing number of nodes is 4 number of nodes after enlargement is 6 1.2 * 24GiB * 6 / 4 = 44GiB 2. Installing the nodes In Preparation for the installation of the new nodes you need to know the MAC addresses and which disk configuration is set on the running part of the cluster. 2.1. Add node Open 'Configuration > Nodes' Click "Add" configure the node to add click apply Repeat that for every node 2.2. Install nodes Open 'Configuration > Nodes' Reload the page until the state of the recently added nodes changes to 'running' Set the status of all installed nodes to "active" At the end is should look like this: 3. Performing the enlargement 3.1. Shutdown all databases Open 'Services > EXASolution' Check the database operations. If the database is stopped while an operation is in progress, the operation will be aborted Select the EXASolution instances Click "Shutdown" Reload the page until all instances change their status from "Running" to "Created" Again, you may follow the procedure in an appropriate logservice: System marked as stopped. Successfully send retry shutdown event to system partition 64. EXASolution exa_db is rejecting connections controller-st(0.0): Shutdown called. User 0 requests shutdown of system. 3.2 Add nodes to storage After the nodes have been installed, they'll be shown automatically in the storage overview Add now the unused disks to the new nodes The new nodes should look like a completely free node Edit the volume by clicking on the name of the persistent volume Click on edit Be sure, the enlarge will be done on the right volume Add the two nodes and apply the configuration The recently added nodes are now shown in the list 3.3 Add nodes to Database Open 'Services > EXASolution' and click on the database instance Click on edit Add the two nodes as reserve nodes to the database Click apply 3.4 Enlarge the database and start reorganize Check point 1.1 again. If the quota is exceeded, the database won't start again Open the dropdown menu on the right side and submit "Enlarge" Type in the number of master nodes and click apply After applying, the database will start itself. Status is "Starting" The status changes to "Running" if the database is up Now login with EXAPlus-Client and start command : REORGANIZE DATABASE ; Command has been started. For appraisal how long the Reorganize will take, feel free to contact the EXASOL support. Reorganize has been done Now the distribution of the disk usage should be approximately the same Additional information In case you want to enlarge with a number of nodes, which isn't a multiple of the redundancy you have chosen for the volume: You need to drop the redundancy before you can add the nodes to the volume. After you have added the nodes you can increase the redundancy again. As a result of dropping and increasing the redundancy, a recovery process will take place. DO NOT continue with the enlargement procedure during the recovery process e.g. It is possible to enlarge without dropping the redundancy, if you add two, four, six, eight, ... (a multiple of two) number of nodes to the cluster if the redundancy is two.
    • Category 1:
      Cluster Administration - Cluster Management
    • Category 2:
      Cluster Administration - DB Instance Management

      Attachments

      1. 1.1.1_checkmaximalvolumesize.png
        1.1.1_checkmaximalvolumesize.png
        58 kB
      2. 1.1.2_exasolutioninstance.png
        1.1.2_exasolutioninstance.png
        100 kB
      3. 1.2.1_checkstoragesize.png
        1.2.1_checkstoragesize.png
        97 kB
      4. 2.1.1_add_node.png
        2.1.1_add_node.png
        92 kB
      5. 2.1.2_configure_node_to_add.png
        2.1.2_configure_node_to_add.png
        96 kB
      6. 2.2.2_installing_nodes.png
        2.2.2_installing_nodes.png
        118 kB
      7. 2.2.3_installing_done.png
        2.2.3_installing_done.png
        108 kB
      8. 2.2.3_set_installed_nodes_to_active.png
        2.2.3_set_installed_nodes_to_active.png
        108 kB
      9. 2.2.4_new_nodes_are_active.png
        2.2.4_new_nodes_are_active.png
        109 kB
      10. 3.1.1_check_database_operations.png
        3.1.1_check_database_operations.png
        59 kB
      11. 3.1.2_shutdown_db.png
        3.1.2_shutdown_db.png
        59 kB
      12. 3.1.3_shutdown_db_2.png
        3.1.3_shutdown_db_2.png
        58 kB
      13. 3.2.1_nodes_without_disks.png
        3.2.1_nodes_without_disks.png
        98 kB
      14. 3.2.2_Add_unused_disks.png
        3.2.2_Add_unused_disks.png
        98 kB
      15. 3.2.3_new_nodes_with_disks.png
        3.2.3_new_nodes_with_disks.png
        98 kB
      16. 3.2.4_volume_overview.png
        3.2.4_volume_overview.png
        82 kB
      17. 3.2.5_edit_volume.png
        3.2.5_edit_volume.png
        64 kB
      18. 3.2.6_save_volume.png
        3.2.6_save_volume.png
        64 kB
      19. 3.2.7_enlarged_volume.png
        3.2.7_enlarged_volume.png
        83 kB
      20. 3.3.1_db_overview_before_add.png
        3.3.1_db_overview_before_add.png
        100 kB
      21. 3.3.2_add_nodes_to_db.png
        3.3.2_add_nodes_to_db.png
        55 kB
      22. 3.3.3_node_addded_to_db.png
        3.3.3_node_addded_to_db.png
        61 kB
      23. 3.3.4_db_overview_with_new_nodes.png
        3.3.4_db_overview_with_new_nodes.png
        104 kB
      24. 3.4.1_choose_enlarge_in_dropdown.png
        3.4.1_choose_enlarge_in_dropdown.png
        104 kB
      25. 3.4.2_enlarge_the_db.png
        3.4.2_enlarge_the_db.png
        47 kB
      26. 3.4.3_db_starts_itself.png
        3.4.3_db_starts_itself.png
        105 kB
      27. 3.4.4_started_db.png
        3.4.4_started_db.png
        103 kB
      28. 3.4.5_start_reorganize.png
        3.4.5_start_reorganize.png
        102 kB
      29. 3.4.5_started_db.png
        3.4.5_started_db.png
        103 kB
      30. 3.4.6_reorganize_running.png
        3.4.6_reorganize_running.png
        188 kB
      31. 3.4.7_reorganize_done.png
        3.4.7_reorganize_done.png
        62 kB
      32. 3.4.8_load_distribution.png
        3.4.8_load_distribution.png
        104 kB
      33. 3.4.9_load_distribution.png
        3.4.9_load_distribution.png
        104 kB

        Issue Links

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: