This troubleshooting guide provides a detailed process to diagnose and resolve SSL/TLS handshake failures due to certificate errors on OpsRamp Gateways. The steps will help you identify the cause of the disconnect, obtain the necessary certificates, and install them on the gateway to restore secure communication.

Tunnel disconnect with certificate errors in Classic Gateway

If you see the following exception in the vprobe-tls-comm.log, it means you need to upload the CA certificate to both Java and the Gateway OS:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Steps to Troubleshoot

To resolve this issue, upload the necessary certificates to the Java and Gateway OS.

Step 1: Varify if the certificate issuer is DigiCert

Run the following command to check whether the certificate issuer is DigiCert (the expected issuer for OpsRamp):

  • Direct Connection:
    openssl s_client -connect {opsramp-api-server}:{port}
  • Proxy Connection:
    openssl s_client -connect "{opsramp-api-server:api-server-port}" -proxy {proxyip}:{proxy-port}
  • Sample Output for OpsRamp Server Certificate:
    Certificate chain
    0 s:CN = *.api.opsramp.com
   i:C = US, O = DigiCert Inc, CN = RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
1 s:C = US, O = DigiCert Inc, CN = RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
  • Sample Output for Non-OpsRamp Server Certificate:
    Certificate chain
    0 s:CN = *.api.opsramp.com
   i:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, OU = ZIS, CN = wclucp.wue.kba.ag
1 s:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, OU = ZIS, CN = wclucp.wue.kba.ag
   i:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, CN = KBA SubCA Wue I4
2 s:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, CN = KBA SubCA Wue I4
   i:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, CN = Koenig und Bauer AG RootCA3
3 s:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, CN = Koenig und Bauer AG RootCA3
   i:C = DE, L = Wuerzburg, O = Koenig und Bauer AG, CN = Koenig und Bauer AG RootCA3

Step 2: Retrieve Non-OpsRamp/Local Issuer Certificate

If the issuer is not DigiCert, retrieve the local issuer certificate using the following command:

  • Direct Connection:
    openssl s_client -connect {opsramp-api-server}:{port} -showcerts
  • Proxy Connection:
    openssl s_client -connect "{opsramp-api-server:api-server-port}" -proxy {proxyip}:{proxy-port} -showcerts
  • Copy the Certificate Output to a file (e.g., /home/ruser/cert1.crt):
    -----BEGIN CERTIFICATE-----
MIIHnTCCBYWgAwIBAgIQApMqHIN6MX/8V6+KRuvlIjANBgkqhkiG9w0BAQsFADBc
MQswCQYDVQQGEwJVUzEXMBUGA1UEChMORGlnaUNlcnQsIEluYy4xNDAyBgNVBAMT
K1JhcGlkU1NMIEdsb2JhbCBUTFMgUlNBNDA5NiBTSEEyNTYgMjAyMiBDQTEwHhcN
MjIwOTI2MDAwMDAwWhcNMjMxMDI1MjM1OTU5WjAcMRowGAYDVQQDDBEqLmFwaS5v
cHNyYW1wLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOYUqmn3
GBFifgvw3d9kCplfUdFtcn8BJDPm2d6cfDqQqNXvJ6sOxoH4pHLaK2Z+AWl+7iqt
hs+OEumFX/0kOsfFVf/E6LVPDEX0z4mjXaNM3I7/qDMG
-----END CERTIFICATE-----

Note: If you have multiple certificates, create multiple files for each certificate.

Step 3: Uploading the Local Issuer Certificate to the vProbe Service and Gateway

  1. Upload the Certificate to the vProbe Service
    • After copying the certificates, use the following command to import each certificate into the Java keystore with a unique alias.
      Example: For two certificate files:
      sudo keytool -import -noprompt -trustcacerts -alias "{AliasName}-1" -file /home/ruser/cert1.crt -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit

sudo keytool -import -noprompt -trustcacerts -alias "{AliasName}-2" -file /home/ruser/cert2.crt -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit
    • Verify the Certificate Upload
      To confirm the certificate was successfully uploaded to the vProbe service, run the following command:
      keytool -list -v -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts | grep -i '{AliasName}'
  2. Upload the certificate to the Gateway
    • Use the following commands as examples:
      sudo cp /home/ruser/cert1.crt /usr/local/share/ca-certificates/
sudo cp /home/ruser/cert2.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Commands:

  1. Extract certificates from a server via a proxy:
    openssl s_client -showcerts -verify 5 -connect {server_name_or_ip}:443 -proxy {proxy_name_or_ip}:{proxy_port} < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){a++}; out="cert"a".pem"; print >out}'
  2. Print certificate subjects from the CA certificate file:
    openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt | openssl pkcs7 -print_certs -noout | grep subject
  3. Connect to a server via a proxy:
    openssl s_client -showcerts -verify 5 -connect {server_name_or_ip}:443 -proxy {proxy_name_or_ip}:{proxy_port}

Useful Link:

Uploading OpenSSL 3.x.x to Connect to Proxy Servers with Authentication on Older Gateways

  1. Download and Extract OpenSSL:
    wget https://opsramp-gateway.s3.us-east-2.amazonaws.com/Gateway+Custom+Files/openssl-309-ubuntu1804.tar.gz
tar -xvzpPf openssl-309-ubuntu1804.tar.gz -C /desired/path/
  2. Navigate to OpenSSL Directory:
    cd /desired/path/usr/local/ssl
export LD_LIBRARY_PATH=/usr/local/ssl/lib64/
  3. Connect to a Target Server via Proxy with Authentication:
    ./bin/openssl s_client -proxy {proxy_ip}:{proxy_port} -proxy_user "{proxy_username}" -proxy_pass "pass:{proxy_password}" -connect "{target_server_ip_or_name}:{target_port}"

Tunnel disconnect with certificate errors in NextGen Gateway

Follow the below steps tp resolve the Tunnel Disconnect Issues Due to Certificate Errors in NextGen Gateway:

Step 1: Save Certificates to /root/custom-certs

  1. Create a Folder:
    mkdir -p /root/custom-certs
  2. Navigate to the Folder:
    cd /root/custom-certs
  3. Save Certificates:
    • Save all the certificates in this folder with a .crt extension
    • Make sure that you save the certificates on all nodes.

Step 2: Import CA Certificates into the OS (Ubuntu)

  1. Copy Certificates:
    sudo cp /root/custom-certs/*.crt /usr/local/share/
  2. Update Certificates:
    sudo update-ca-certificates

Note: Repeat this process on all nodes.

Step 3: Import CA Certificates into NextGen Gateway

  1. Create a ConfigMap:
    cd /root/
kubectl create configmap custom-certs --from-file=custom-certs -n <namespace>
  2. Update the Helm Chart:
    • If the gateway is already registered, pull and update the Helm chart
      helm pull oci://us-docker.pkg.dev/opsramp-registry/gateway-cluster-charts/nextgen-gw --version 1.6.0
helm upgrade nextgen-gw nextgen-gw-1.6.0.tgz -n <namespace>
    • Note: The –version 1.6.0 and nextgen-gw-1.6.0.tgz should match your specific gateway release.
  3. Register the Gateway (if not registered):
    Follow the registration document if the gateway is not already registered.

Step 4: Update Existing Certificates

  1. Delete the Existing ConfigMap:
    kubectl delete configmap custom-certs -n <namespace>
  2. Re-import the Certificates:
    Follow Step 3 and the steps under “Import CA Certificates into the OS (Ubuntu)”.