Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-87

Main Memory Management and Performance Indications

    Details

    • Type: Explanation
    • Status: Published
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: EXASolution
    • Labels:
    • Explanation:
      Hide

      Background

      You always get the highest performance if the data to be analyzed fits into main memory available to the database (your license).

      Explanation

      EXASolution stores data column-wise both in memory and on disk. Each column is spread across a number of physical blocks and of course across the nodes. Only blocks, required to compute the query, will be loaded into main memory.

      Indexes will be treated in the same way: indexes are physically stored in blocks and only the needed blocks will be loaded into main memory. Please refer to SOL-6 for more details on indexes.

      Typically, there's no need to hold the full set of data in memory. EXASolution loads the data requested by queries into main memory on demand (so called hot data) and swapps them out if needed. EXASolution monitors the memory consumption and provides you with a suggestion for the optimum sizing (RECOMMENDED_DB_RAM_SIZE_*).

      If you persistently observe, that "RECOMMENDED_DB_RAM_SIZE_AVG" is at least 30 to 50% higher than your license and you experience performance issues, it means you would need to take an increase of DBRAM into consideration. On the other hand, a deeper system analysis could help to identify queries causing high memory consumption. There are a number of ways for improving the performance and for reducing the memory consumption in EXASolution.

      In general, upgrading hardware/RAM or license should be bottleneck driven. To increase performance or it has measurably deterred over time, than we suggest to investigate the bottleneck and work on this by

      • identifying critical queries and optimize them
      • adding more nodes if your system is CPU bound
      • adding more disks/nodes if the space runs out anyway
      • adding a better network if your system is network bound
      • increasing DBRAM if your system has lots of HDD_READ

      Additional References

      Please consider becoming an EXASOL Certified Performance Expert by attending our Performance Training: see http://www.exasol.com/en/services/training/performance-expert/

      Show
      Background You always get the highest performance if the data to be analyzed fits into main memory available to the database (your license). Explanation EXASolution stores data column-wise both in memory and on disk. Each column is spread across a number of physical blocks and of course across the nodes. Only blocks, required to compute the query, will be loaded into main memory. Indexes will be treated in the same way: indexes are physically stored in blocks and only the needed blocks will be loaded into main memory. Please refer to SOL-6 for more details on indexes. Typically, there's no need to hold the full set of data in memory. EXASolution loads the data requested by queries into main memory on demand (so called hot data) and swapps them out if needed. EXASolution monitors the memory consumption and provides you with a suggestion for the optimum sizing (RECOMMENDED_DB_RAM_SIZE_*). If you persistently observe, that "RECOMMENDED_DB_RAM_SIZE_AVG" is at least 30 to 50% higher than your license and you experience performance issues, it means you would need to take an increase of DBRAM into consideration. On the other hand, a deeper system analysis could help to identify queries causing high memory consumption. There are a number of ways for improving the performance and for reducing the memory consumption in EXASolution. In general, upgrading hardware/RAM or license should be bottleneck driven. To increase performance or it has measurably deterred over time, than we suggest to investigate the bottleneck and work on this by identifying critical queries and optimize them adding more nodes if your system is CPU bound adding more disks/nodes if the space runs out anyway adding a better network if your system is network bound increasing DBRAM if your system has lots of HDD_READ Additional References Please consider becoming an EXASOL Certified Performance Expert by attending our Performance Training: see http://www.exasol.com/en/services/training/performance-expert/
    • Category 1:
      Database Administration - Performance

      Attachments

        Issue Links

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: