Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-559

UDF to synchronize Archive Volumes via FTP

    Details

    • Type: How To
    • Status: Published
    • Affects Version/s: EXASOL 6.0.0, EXASolution 5.0.0
    • Fix Version/s: None
    • Component/s: EXASolution
    • Labels:
    • Solution:
      Hide

      With this UDF (backup_copy_ftp.sql) written in Python, you can easily synchronize archive volumes between clusters. Transport is TLS encrypted (self._ftp = FTP_TLS). After volumes have been initially synchronized all files added or deleted will be added or deleted in the target archive volume. This UDF does not support synchronizing specific days or backup IDs but it can be easily adjusted to your needs. Parallelity is handled by the database, for best performance, the number of database and master nodes of the target archive volume should be the same.

      If a synchronization attempt fails cleanup target volume manually

      Usage:

      Within the UDF adjust these variables accordingly:

      LOCAL_URL    = 'ftp://ExaoperationUser:EXAoperationPW@%s/SourceArchiveVolumeID'
      REMOTE_URL   = 'ftp://EXAoperationUser:EXAoperationPW@%s/TargetArchiveVolumeID'
      REMOTE_NODES = [ 'IP node 11', 'IP node 12', 'IP node 13']
      
      -- To use it, simply set the correct access URLs and IP addresses to
      -- the remote nodes, create the UDF and call it in following SQL:
      -- 
      -- SQL_EXA> SELECT syncBackups(IPROC) FROM EXA_LOADAVG;
      --
      -- The copy process runs completely parallel, distributed over all
      -- nodes and only files missing in the remote system are copied, so
      -- this UDF can be called on regularly. Backup files which are removed
      -- from the source will also be removed in the remote system.
      
      Show
      With this UDF (backup_copy_ftp.sql) written in Python, you can easily synchronize archive volumes between clusters. Transport is TLS encrypted (self._ftp = FTP_TLS). After volumes have been initially synchronized all files added or deleted will be added or deleted in the target archive volume. This UDF does not support synchronizing specific days or backup IDs but it can be easily adjusted to your needs. Parallelity is handled by the database, for best performance, the number of database and master nodes of the target archive volume should be the same. If a synchronization attempt fails cleanup target volume manually Usage: Within the UDF adjust these variables accordingly: LOCAL_URL = 'ftp://ExaoperationUser:EXAoperationPW@%s/SourceArchiveVolumeID' REMOTE_URL = 'ftp://EXAoperationUser:EXAoperationPW@%s/TargetArchiveVolumeID' REMOTE_NODES = [ 'IP node 11', 'IP node 12', 'IP node 13'] -- To use it, simply set the correct access URLs and IP addresses to -- the remote nodes, create the UDF and call it in following SQL: -- -- SQL_EXA> SELECT syncBackups(IPROC) FROM EXA_LOADAVG; -- -- The copy process runs completely parallel, distributed over all -- nodes and only files missing in the remote system are copied, so -- this UDF can be called on regularly. Backup files which are removed -- from the source will also be removed in the remote system.
    • Category 1:
      Cluster Administration - Backup

      Attachments

        Activity

          People

          • Assignee:
            CaptainEXA Captain EXASOL
            Reporter:
            CaptainEXA Captain EXASOL
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: