Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-261

Cluster Enlargement: Adding Additional Nodes

    Details

    • Type: How To
    • Status: Published
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: EXASolution, EXAStorage
    • Labels:
      None
    • Solution:
      Hide

      Synopsis

      This article guides through the enlargement of a cluster by way of example. It shows how to enlarge a 4-node system with data volume redundancy 2 by 2 additional nodes.

      Please note that

      1. The tasks performed in EXAoperation require a user with at least "Administrator" privileges.
      2. It is advisable to have a recent backup available offline (stored outside of the cluster).
      3. The "REORGANIZE DATABASE" command may be executed while other clients connect but it is recommended to execute it as exclusive task.
      4. We urge to either remove any database quota (Maximal Volume Size) during the enlargement/reorganize or to set it to a high enough value (see section 'Preparations' below).


      1. Preparations

      1.1. Review database instance quota and active nodes
      • Open 'Services > EXASolution'
      • Open the EXASolution instance by clicking on instance name

        Please note the number of active nodes for that database instance and the value in field Maximal Volume Size (GiB) and (an empty field indicates that there is no quota set).
      1.2. Review the volume size
      • Open 'Services > EXAStorage'
      • Note the size of the persistent volume.
      1.3. Calculations

      Database quotas limit the overall data volume (persistent & temporary tablespace) that may be allocated by an EXASolution instance. Redundancy is not taken into account.

      The quota must reflect the disk space requirements after the enlargement:

      1. The reorganize is a demanding operation and may trigger swapping out of blocks of data. We hence recommend to allow for some reserve (20% in this calculation).
      2. The quota must be bigger than the enlarged volume plus some overhead.
      1.2 * persistent volume size * number of nodes after enlargement / existing number of nodes 
      

      In our example:

      • volume size is 24GiB
      • existing number of nodes is 4
      • number of nodes after enlargement is 6
        1.2 * 24GiB * 6 / 4 = 44GiB

      2. Installing the nodes

      In Preparation for the installation of the new nodes you need to know the MAC addresses and which disk configuration is set on the running part of the cluster.

      2.1. Add node
      • Open 'Configuration > Nodes'
      • Click "Add"
        • configure the node to add
        • click apply


      • Repeat that for every node
      2.2. Install nodes
      • Open 'Configuration > Nodes'
      • Reload the page until the state of the recently added nodes changes to 'running'
      • Set the status of all installed nodes to "active"

      At the end is should look like this:

      3. Performing the enlargement

      3.1. Shutdown all databases
      • Open 'Services > EXASolution'
      • Check the database operations. If the database is stopped while an operation is in progress, the operation will be aborted
      • Select the EXASolution instances
      • Click "Shutdown"
      • Reload the page until all instances change their status from "Running" to "Created"
      • Again, you may follow the procedure in an appropriate logservice:

        System marked as stopped.
        Successfully send retry shutdown event to system partition 64.
        EXASolution exa_db is rejecting connections
        controller-st(0.0): Shutdown called.
        User 0 requests shutdown of system.

      3.2 Add nodes to storage
      • After the nodes have been installed, they'll be shown automatically in the storage overview
      • Add now the unused disks to the new nodes
      • The new nodes should look like a completely free node
      • Edit the volume by clicking on the name of the persistent volume
      • Click on edit
        Be sure, the enlarge will be done on the right volume
      • Add the two nodes and apply the configuration
      • The recently added nodes are now shown in the list
      3.3 Add nodes to Database
      • Open 'Services > EXASolution' and click on the database instance
      • Click on edit
      • Add the two nodes as reserve nodes to the database
      • Click apply
      3.4 Enlarge the database and start reorganize

      Check point 1.1 again. If the quota is exceeded, the database won't start again

      • Open the dropdown menu on the right side and submit "Enlarge"
      • Type in the number of master nodes and click apply
      • After applying, the database will start itself. Status is "Starting"
      • The status changes to "Running" if the database is up
      • Now login with EXAPlus-Client and start command :
         REORGANIZE DATABASE; 

      • Command has been started. For appraisal how long the Reorganize will take, feel free to contact the EXASOL support.
      • Reorganize has been done
      • Now the distribution of the disk usage should be approximately the same



      Additional information
      In case you want to enlarge with a number of nodes, which isn't a multiple of the redundancy you have chosen for the volume:
      You need to drop the redundancy before you can add the nodes to the volume. After you have added the nodes you can increase the redundancy again.
      As a result of dropping and increasing the redundancy, a recovery process will take place. DO NOT continue with the enlargement procedure during the recovery process
      e.g. It is possible to enlarge without dropping the redundancy, if you add two, four, six, eight, ... (a multiple of two) number of nodes to the cluster if the redundancy is two.

      Show
      Synopsis This article guides through the enlargement of a cluster by way of example. It shows how to enlarge a 4-node system with data volume redundancy 2 by 2 additional nodes. Please note that The tasks performed in EXAoperation require a user with at least "Administrator" privileges. It is advisable to have a recent backup available offline (stored outside of the cluster). The "REORGANIZE DATABASE" command may be executed while other clients connect but it is recommended to execute it as exclusive task. We urge to either remove any database quota (Maximal Volume Size) during the enlargement/reorganize or to set it to a high enough value (see section 'Preparations' below). 1. Preparations 1.1. Review database instance quota and active nodes Open 'Services > EXASolution' Open the EXASolution instance by clicking on instance name Please note the number of active nodes for that database instance and the value in field Maximal Volume Size (GiB) and (an empty field indicates that there is no quota set). 1.2. Review the volume size Open 'Services > EXAStorage' Note the size of the persistent volume. 1.3. Calculations Database quotas limit the overall data volume (persistent & temporary tablespace) that may be allocated by an EXASolution instance. Redundancy is not taken into account. The quota must reflect the disk space requirements after the enlargement: The reorganize is a demanding operation and may trigger swapping out of blocks of data. We hence recommend to allow for some reserve (20% in this calculation). The quota must be bigger than the enlarged volume plus some overhead. 1.2 * persistent volume size * number of nodes after enlargement / existing number of nodes In our example: volume size is 24GiB existing number of nodes is 4 number of nodes after enlargement is 6 1.2 * 24GiB * 6 / 4 = 44GiB 2. Installing the nodes In Preparation for the installation of the new nodes you need to know the MAC addresses and which disk configuration is set on the running part of the cluster. 2.1. Add node Open 'Configuration > Nodes' Click "Add" configure the node to add click apply Repeat that for every node 2.2. Install nodes Open 'Configuration > Nodes' Reload the page until the state of the recently added nodes changes to 'running' Set the status of all installed nodes to "active" At the end is should look like this: 3. Performing the enlargement 3.1. Shutdown all databases Open 'Services > EXASolution' Check the database operations. If the database is stopped while an operation is in progress, the operation will be aborted Select the EXASolution instances Click "Shutdown" Reload the page until all instances change their status from "Running" to "Created" Again, you may follow the procedure in an appropriate logservice: System marked as stopped. Successfully send retry shutdown event to system partition 64. EXASolution exa_db is rejecting connections controller-st(0.0): Shutdown called. User 0 requests shutdown of system. 3.2 Add nodes to storage After the nodes have been installed, they'll be shown automatically in the storage overview Add now the unused disks to the new nodes The new nodes should look like a completely free node Edit the volume by clicking on the name of the persistent volume Click on edit Be sure, the enlarge will be done on the right volume Add the two nodes and apply the configuration The recently added nodes are now shown in the list 3.3 Add nodes to Database Open 'Services > EXASolution' and click on the database instance Click on edit Add the two nodes as reserve nodes to the database Click apply 3.4 Enlarge the database and start reorganize Check point 1.1 again. If the quota is exceeded, the database won't start again Open the dropdown menu on the right side and submit "Enlarge" Type in the number of master nodes and click apply After applying, the database will start itself. Status is "Starting" The status changes to "Running" if the database is up Now login with EXAPlus-Client and start command : REORGANIZE DATABASE ; Command has been started. For appraisal how long the Reorganize will take, feel free to contact the EXASOL support. Reorganize has been done Now the distribution of the disk usage should be approximately the same Additional information In case you want to enlarge with a number of nodes, which isn't a multiple of the redundancy you have chosen for the volume: You need to drop the redundancy before you can add the nodes to the volume. After you have added the nodes you can increase the redundancy again. As a result of dropping and increasing the redundancy, a recovery process will take place. DO NOT continue with the enlargement procedure during the recovery process e.g. It is possible to enlarge without dropping the redundancy, if you add two, four, six, eight, ... (a multiple of two) number of nodes to the cluster if the redundancy is two.
    • Category 1:
      Cluster Administration - Cluster Management
    • Category 2:
      Cluster Administration - DB Instance Management

      Attachments

        Issue Links

        1. 1.1.1_checkmaximalvolumesize.png
          1.1.1_checkmaximalvolumesize.png
          58 kB
        2. 1.1.2_exasolutioninstance.png
          1.1.2_exasolutioninstance.png
          100 kB
        3. 1.2.1_checkstoragesize.png
          1.2.1_checkstoragesize.png
          97 kB
        4. 2.1.1_add_node.png
          2.1.1_add_node.png
          92 kB
        5. 2.1.2_configure_node_to_add.png
          2.1.2_configure_node_to_add.png
          96 kB
        6. 2.2.2_installing_nodes.png
          2.2.2_installing_nodes.png
          118 kB
        7. 2.2.3_installing_done.png
          2.2.3_installing_done.png
          108 kB
        8. 2.2.3_set_installed_nodes_to_active.png
          2.2.3_set_installed_nodes_to_active.png
          108 kB
        9. 2.2.4_new_nodes_are_active.png
          2.2.4_new_nodes_are_active.png
          109 kB
        10. 3.1.1_check_database_operations.png
          3.1.1_check_database_operations.png
          59 kB
        11. 3.1.2_shutdown_db.png
          3.1.2_shutdown_db.png
          59 kB
        12. 3.1.3_shutdown_db_2.png
          3.1.3_shutdown_db_2.png
          58 kB
        13. 3.2.1_nodes_without_disks.png
          3.2.1_nodes_without_disks.png
          98 kB
        14. 3.2.2_Add_unused_disks.png
          3.2.2_Add_unused_disks.png
          98 kB
        15. 3.2.3_new_nodes_with_disks.png
          3.2.3_new_nodes_with_disks.png
          98 kB
        16. 3.2.4_volume_overview.png
          3.2.4_volume_overview.png
          82 kB
        17. 3.2.5_edit_volume.png
          3.2.5_edit_volume.png
          64 kB
        18. 3.2.6_save_volume.png
          3.2.6_save_volume.png
          64 kB
        19. 3.2.7_enlarged_volume.png
          3.2.7_enlarged_volume.png
          83 kB
        20. 3.3.1_db_overview_before_add.png
          3.3.1_db_overview_before_add.png
          100 kB
        21. 3.3.2_add_nodes_to_db.png
          3.3.2_add_nodes_to_db.png
          55 kB
        22. 3.3.3_node_addded_to_db.png
          3.3.3_node_addded_to_db.png
          61 kB
        23. 3.3.4_db_overview_with_new_nodes.png
          3.3.4_db_overview_with_new_nodes.png
          104 kB
        24. 3.4.1_choose_enlarge_in_dropdown.png
          3.4.1_choose_enlarge_in_dropdown.png
          104 kB
        25. 3.4.2_enlarge_the_db.png
          3.4.2_enlarge_the_db.png
          47 kB
        26. 3.4.3_db_starts_itself.png
          3.4.3_db_starts_itself.png
          105 kB
        27. 3.4.4_started_db.png
          3.4.4_started_db.png
          103 kB
        28. 3.4.5_start_reorganize.png
          3.4.5_start_reorganize.png
          102 kB
        29. 3.4.5_started_db.png
          3.4.5_started_db.png
          103 kB
        30. 3.4.6_reorganize_running.png
          3.4.6_reorganize_running.png
          188 kB
        31. 3.4.7_reorganize_done.png
          3.4.7_reorganize_done.png
          62 kB
        32. 3.4.8_load_distribution.png
          3.4.8_load_distribution.png
          104 kB
        33. 3.4.9_load_distribution.png
          3.4.9_load_distribution.png
          104 kB

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: