[SOL-87] Main Memory Management and Performance Indications Created: 21.05.2014  Updated: 26.03.2020

Status: Published
Project: Solution Center
Component/s: EXASolution
Affects Version/s: None
Fix Version/s: None

Type: Explanation
Reporter: Captain EXASOL Assignee: Captain EXASOL
Labels: In-Memory

Issue Links:
Causing
Related
relates SOL-6 Indexes Published
Explanation:

Background

You always get the highest performance if the data to be analyzed fits into main memory available to the database (your license).

Explanation

EXASolution stores data column-wise both in memory and on disk. Each column is spread across a number of physical blocks and of course across the nodes. Only blocks, required to compute the query, will be loaded into main memory.

Indexes will be treated in the same way: indexes are physically stored in blocks and only the needed blocks will be loaded into main memory. Please refer to SOL-6 for more details on indexes.

Typically, there's no need to hold the full set of data in memory. EXASolution loads the data requested by queries into main memory on demand (so called hot data) and swapps them out if needed. EXASolution monitors the memory consumption and provides you with a suggestion for the optimum sizing (RECOMMENDED_DB_RAM_SIZE_*).

If you persistently observe, that "RECOMMENDED_DB_RAM_SIZE_AVG" is at least 30 to 50% higher than your license and you experience performance issues, it means you would need to take an increase of DBRAM into consideration. On the other hand, a deeper system analysis could help to identify queries causing high memory consumption. There are a number of ways for improving the performance and for reducing the memory consumption in EXASolution.

In general, upgrading hardware/RAM or license should be bottleneck driven. To increase performance or it has measurably deterred over time, than we suggest to investigate the bottleneck and work on this by

  • identifying critical queries and optimize them
  • adding more nodes if your system is CPU bound
  • adding more disks/nodes if the space runs out anyway
  • adding a better network if your system is network bound
  • increasing DBRAM if your system has lots of HDD_READ

Additional References

Please consider becoming an EXASOL Certified Performance Expert by attending our Performance Training: see http://www.exasol.com/en/services/training/performance-expert/

Category 1: Database Administration - Performance

 Comments   
Comment by Peggy Schmidt-Mittenzwei [ 26.03.2020 ]

Altered to Background - Explanation - template

Generated at Sun Jul 12 15:00:32 CEST 2020 using Jira 7.13.13#713013-sha1:4c0f5f5e3383570393fbbf3d2fd5de1eb4057c36.