Uploaded image for project: 'Solution Center'
  1. Solution Center
  2. SOL-325

Testing HTTP connections using python UDF

    XMLWordPrintable

    Details

    • Type: How To
    • Status: Published
    • Affects Version/s: EXASolution 5.0, EXASOL 6.0.0, Exasol 6.1.0
    • Fix Version/s: None
    • Component/s: EXAPowerlytics
    • Labels:
      None
    • Solution:
      Hide

      Preface

      As EXASolution comes with its own operating system and users do rarely have access to that system, testing network connectivity can be quite hard. This makes debugging Import/Export/UDF that fail with timeouts or connection refused a lengthy task.

      Solution

      The attached SQL file contains a SCALAR PYTHON SCRIPT that does minimalistic network testing using the following steps:

      1. Hostname lookup using DNS (Domain Name Service)
      2. TCP connect to the resulting address
      3. HTTP 1.1 request

      The script will output some information, including the cluster node it ran on and an elapsed time (in seconds).

      Example usage

      1 - Testing an internal web server on a random node

      select webtest_py('192.168.103.212', 80, '/')
      order by 1,2;
      
      NODE_IP SECONDS MESSAGE
      n0016.c0001.exacluster.local 0.009 Checking DNS for 192.168.103.212
      n0016.c0001.exacluster.local 0.009 Name resolves to IPs ['192.168.103.212']
      n0016.c0001.exacluster.local 0.009 Trying to connect to 192.168.103.212, port 80
      n0016.c0001.exacluster.local 0.010 Connected.
      n0016.c0001.exacluster.local 0.010 HTTP GET request sent
      n0016.c0001.exacluster.local 0.011 HTTP/1.1 200 OK
      n0016.c0001.exacluster.local 0.011 ...Date: Mon, 12 Oct 2015 14:37:19 GMT
      n0016.c0001.exacluster.local 0.011 ...Server: Apache/2.2.9 (Debian) mod_auth_kerb/5.3 DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny16 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
      n0016.c0001.exacluster.local 0.011 ...Last-Modified: Mon, 16 Jun 2008 08:57:00 GMT
      n0016.c0001.exacluster.local 0.011 ...ETag: "50800c-128-44fc4cf7b6f00"
      n0016.c0001.exacluster.local 0.011 ...Accept-Ranges: bytes
      n0016.c0001.exacluster.local 0.011 ...Content-Length: 296
      n0016.c0001.exacluster.local 0.011 ...Vary: Accept-Encoding
      n0016.c0001.exacluster.local 0.011 ...Content-Type: text/html
      n0016.c0001.exacluster.local 0.011 End of headers

      2 - Invalid page

      select webtest_py('192.168.103.212', 80, '/xxx.html')
      order by 1,2;
      
      NODE_IP SECONDS MESSAGE
      n0016.c0001.exacluster.local 0.002 Checking DNS for 192.168.103.212
      n0016.c0001.exacluster.local 0.002 Name resolves to IPs ['192.168.103.212']
      n0016.c0001.exacluster.local 0.003 Trying to connect to 192.168.103.212, port 80
      n0016.c0001.exacluster.local 0.003 Connected.
      n0016.c0001.exacluster.local 0.003 HTTP GET request sent
      n0016.c0001.exacluster.local 0.004 HTTP/1.1 404 Not Found
      n0016.c0001.exacluster.local 0.004 ...Date: Mon, 12 Oct 2015 14:50:31 GMT
      n0016.c0001.exacluster.local 0.005 ...Server: Apache/2.2.9 (Debian) mod_auth_kerb/5.3 DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny16 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
      n0016.c0001.exacluster.local 0.005 ...Vary: Accept-Encoding
      n0016.c0001.exacluster.local 0.005 ...Content-Length: 388
      n0016.c0001.exacluster.local 0.005 ...Content-Type: text/html; charset=iso-8859-1
      n0016.c0001.exacluster.local 0.005 End of headers

      3 - Trying wrong port:

      select webtest_py('192.168.103.212', 20, '/')
      order by 1,2;
      
      NODE_IP SECONDS MESSAGE
      n0016.c0001.exacluster.local 0.013 Checking DNS for 192.168.103.212
      n0016.c0001.exacluster.local 0.013 Name resolves to IPs ['192.168.103.212']
      n0016.c0001.exacluster.local 0.014 Trying to connect to 192.168.103.212, port 20
      n0016.c0001.exacluster.local 0.014 Failed: [Errno 111] Connection refused

      4 - Firewall blocks external connection from all nodes:

      select webtest_py('www.heise.de', 80, '/')
      from exa_loadavg
      order by 1,2;
      
      NODE_IP SECONDS MESSAGE
      n0016.c0001.exacluster.local 0.002 Checking DNS for www.heise.de
      n0016.c0001.exacluster.local 0.003 Name resolves to IPs ['193.99.144.85']
      n0016.c0001.exacluster.local 0.004 Trying to connect to 193.99.144.85, port 80
      n0016.c0001.exacluster.local 63.004 Failed: [Errno 110] Connection timed out
      n0017.c0001.exacluster.local 0.009 Checking DNS for www.heise.de
      n0017.c0001.exacluster.local 0.011 Name resolves to IPs ['193.99.144.85']
      n0017.c0001.exacluster.local 0.011 Trying to connect to 193.99.144.85, port 80
      n0017.c0001.exacluster.local 63.011 Failed: [Errno 110] Connection timed out
      n0019.c0001.exacluster.local 0.009 Checking DNS for www.heise.de
      n0019.c0001.exacluster.local 0.011 Name resolves to IPs ['193.99.144.85']
      n0019.c0001.exacluster.local 0.011 Trying to connect to 193.99.144.85, port 80
      n0019.c0001.exacluster.local 63.011 Failed: [Errno 110] Connection timed out
      n0020.c0001.exacluster.local 0.002 Checking DNS for www.heise.de
      n0020.c0001.exacluster.local 0.003 Name resolves to IPs ['193.99.144.85']
      n0020.c0001.exacluster.local 0.004 Trying to connect to 193.99.144.85, port 80
      n0020.c0001.exacluster.local 63.003 Failed: [Errno 110] Connection timed out
      Show
      Preface As EXASolution comes with its own operating system and users do rarely have access to that system, testing network connectivity can be quite hard. This makes debugging Import/Export/UDF that fail with timeouts or connection refused a lengthy task. Solution The attached SQL file contains a SCALAR PYTHON SCRIPT that does minimalistic network testing using the following steps: Hostname lookup using DNS (Domain Name Service) TCP connect to the resulting address HTTP 1.1 request The script will output some information, including the cluster node it ran on and an elapsed time (in seconds). Example usage 1 - Testing an internal web server on a random node select webtest_py( '192.168.103.212' , 80, '/' ) order by 1,2; NODE_IP SECONDS MESSAGE n0016.c0001.exacluster.local 0.009 Checking DNS for 192.168.103.212 n0016.c0001.exacluster.local 0.009 Name resolves to IPs ['192.168.103.212'] n0016.c0001.exacluster.local 0.009 Trying to connect to 192.168.103.212, port 80 n0016.c0001.exacluster.local 0.010 Connected. n0016.c0001.exacluster.local 0.010 HTTP GET request sent n0016.c0001.exacluster.local 0.011 HTTP/1.1 200 OK n0016.c0001.exacluster.local 0.011 ...Date: Mon, 12 Oct 2015 14:37:19 GMT n0016.c0001.exacluster.local 0.011 ...Server: Apache/2.2.9 (Debian) mod_auth_kerb/5.3 DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny16 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g n0016.c0001.exacluster.local 0.011 ...Last-Modified: Mon, 16 Jun 2008 08:57:00 GMT n0016.c0001.exacluster.local 0.011 ...ETag: "50800c-128-44fc4cf7b6f00" n0016.c0001.exacluster.local 0.011 ...Accept-Ranges: bytes n0016.c0001.exacluster.local 0.011 ...Content-Length: 296 n0016.c0001.exacluster.local 0.011 ...Vary: Accept-Encoding n0016.c0001.exacluster.local 0.011 ...Content-Type: text/html n0016.c0001.exacluster.local 0.011 End of headers 2 - Invalid page select webtest_py( '192.168.103.212' , 80, '/xxx.html' ) order by 1,2; NODE_IP SECONDS MESSAGE n0016.c0001.exacluster.local 0.002 Checking DNS for 192.168.103.212 n0016.c0001.exacluster.local 0.002 Name resolves to IPs ['192.168.103.212'] n0016.c0001.exacluster.local 0.003 Trying to connect to 192.168.103.212, port 80 n0016.c0001.exacluster.local 0.003 Connected. n0016.c0001.exacluster.local 0.003 HTTP GET request sent n0016.c0001.exacluster.local 0.004 HTTP/1.1 404 Not Found n0016.c0001.exacluster.local 0.004 ...Date: Mon, 12 Oct 2015 14:50:31 GMT n0016.c0001.exacluster.local 0.005 ...Server: Apache/2.2.9 (Debian) mod_auth_kerb/5.3 DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny16 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g n0016.c0001.exacluster.local 0.005 ...Vary: Accept-Encoding n0016.c0001.exacluster.local 0.005 ...Content-Length: 388 n0016.c0001.exacluster.local 0.005 ...Content-Type: text/html; charset=iso-8859-1 n0016.c0001.exacluster.local 0.005 End of headers 3 - Trying wrong port: select webtest_py( '192.168.103.212' , 20, '/' ) order by 1,2; NODE_IP SECONDS MESSAGE n0016.c0001.exacluster.local 0.013 Checking DNS for 192.168.103.212 n0016.c0001.exacluster.local 0.013 Name resolves to IPs ['192.168.103.212'] n0016.c0001.exacluster.local 0.014 Trying to connect to 192.168.103.212, port 20 n0016.c0001.exacluster.local 0.014 Failed: [Errno 111] Connection refused 4 - Firewall blocks external connection from all nodes: select webtest_py( 'www.heise.de' , 80, '/' ) from exa_loadavg order by 1,2; NODE_IP SECONDS MESSAGE n0016.c0001.exacluster.local 0.002 Checking DNS for www.heise.de n0016.c0001.exacluster.local 0.003 Name resolves to IPs ['193.99.144.85'] n0016.c0001.exacluster.local 0.004 Trying to connect to 193.99.144.85, port 80 n0016.c0001.exacluster.local 63.004 Failed: [Errno 110] Connection timed out n0017.c0001.exacluster.local 0.009 Checking DNS for www.heise.de n0017.c0001.exacluster.local 0.011 Name resolves to IPs ['193.99.144.85'] n0017.c0001.exacluster.local 0.011 Trying to connect to 193.99.144.85, port 80 n0017.c0001.exacluster.local 63.011 Failed: [Errno 110] Connection timed out n0019.c0001.exacluster.local 0.009 Checking DNS for www.heise.de n0019.c0001.exacluster.local 0.011 Name resolves to IPs ['193.99.144.85'] n0019.c0001.exacluster.local 0.011 Trying to connect to 193.99.144.85, port 80 n0019.c0001.exacluster.local 63.011 Failed: [Errno 110] Connection timed out n0020.c0001.exacluster.local 0.002 Checking DNS for www.heise.de n0020.c0001.exacluster.local 0.003 Name resolves to IPs ['193.99.144.85'] n0020.c0001.exacluster.local 0.004 Trying to connect to 193.99.144.85, port 80 n0020.c0001.exacluster.local 63.003 Failed: [Errno 110] Connection timed out
    • Category 1:
      UDFs and In-Database Analytics

      Attachments

        Issue Links

          Activity

            People

            • Assignee:
              CaptainEXA Captain EXASOL
              Reporter:
              CaptainEXA Captain EXASOL
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated: