Nextgen Gateway Connectivity Issues

Gateway registration establishes a secure and authenticated connection between the OpsRamp platform and your on-premises environment – The Gateway tunnel. Successful registration is required to create and maintain the secured Gateway tunnel. Issues with registration or improper credentials can lead to Transport Layer Security (TLS) handshake failures, resulting in disconnects. Follow these steps to ensure the Gateway register

Unable to Register the Gateway

Verify OpsRamp API Server and Activation Token
- Login to OpsRamp Portal.
- Navigate to Setup > Resources > Management Profiles.
- Choose the required management profile from the Management Profiles section.
- Verify the following registration details:
  - opsramp_apiserver
  - AUTHENTICATION_TOKEN
- Ensure that the details provided on the Gateway webUI registration page match the selected management profile.
- Avoid trailing and leading spaces in the input fields.
If you are unable to register the gateway, run the following commands to check if the cloud url is reachable from node.
- Check API Server Connectivity:
  - Ensure the OpsRamp API server (copied from Step 1) is reachable on port 443. Run the following command in the gateway CLI:
```
telnet <opsramp_api_server> 443
```
- Verify SSL Connection:
  - For a direct connection:
```
openssl s_client -connect <opsramp_api>:443
```
  - For a proxy connection:
```
openssl s_client -connect "<opsramp_api>:443" -proxy <proxy_ip>:<proxy_port>
```
- Ensure OpsRamp IPs are whitelisted. Refer Public IP Addresses document. If required IPs are not listed, contact the OpsRamp SaaS team to the following:

Invoke the getClientByAccessToken API call using curl to ensure proper data is received with the relevant response code.

Copy the AUTHENTICATION_TOKEN from the step 1.

Go the gateway cli and run the following command:

Direct

curl -v --request POST 'https://{{SERVER_URL}}/saregandlicdownload.do?action=getClientByAccessToken' --header 'Content-Type: application/xml' --data-raw '<activation_details action="getClientByAccessToken" type="cpe"><activation_token>{{AUTHENTICATION_TOKEN}}</activation_token></activation_details>'

Proxy without username and password

curl -v --proxy http://{{PROXY_IP}}:{{PROXY_PORT}} --request POST 'https://{{SERVER_URL}}/saregandlicdownload.do?action=getClientByAccessToken' --header 'Content-Type: application/xml' --data-raw '<activation_details action="getClientByAccessToken" type="cpe"><activation_token>{{AUTHENTICATION_TOKEN}}</activation_token></activation_details>'

Proxy with username and password

curl -v --proxy http://{{PROXY_IP}}:{{PROXY_PORT}} --proxy-user {{PROXY_USERNAME}}:{{PROXY_PASSWORD}} --request POST 'https://{{SERVER_URL}}/saregandlicdownload.do?action=getClientByAccessToken' --header 'Content-Type: application/xml' --data-raw '<activation_details action="getClientByAccessToken" type="cpe"><activation_token>{{AUTHENTICATION_TOKEN}}</activation_token></activation_details>'

Gateway Tunnel not up after Gateway registration

Retrieve API Server Details with the following command.

kubectl get secrets vprobe-secret -o jsonpath="{['data']['regInfo']}" | base64 -d

Query the Request Token: Copy the API_KEY and API_SECRET from above command output and replace {api_key_xxxxxxxxxxx} with API_KEY value and {xxxxxxxxxxxxxxx} with CLIENT_SECRET to below url.

Direct

curl -vvv -i L https://{apiServer}/auth/oauth/token -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -d "grant_type=client_credentials&client_id={api_key_xxxxxxxxxxx}&client_secret={xxxxxxxxxxxxxxx}" -X POST

Proxy without username and password

curl -vvv  --proxy http://{proxyIp}:{proxyport} -d "grant_type=client_credentials&client_id={api_key_xxxxxxxxxxx}&client_secret={xxxxxxxxxxxxxxx}" -X POST https://{apiServer}/auth/oauth/token -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json"

Proxy with username and password

curl -vvv  --proxy http://{proxyIp}:{proxyport} --proxy-user {username}:{password} -d "grant_type=client_credentials&client_id={api_key_xxxxxxxxxxx}&client_secret={xxxxxxxxxxxxxxx}" -X POST https://{apiServer}/auth/oauth/token -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json"

Query API Call to Pull csnode Details.

Use the bearer token from the previous response:

Direct

curl -vvv -i https://{apiServer}/api/v2//gateway/connectionNode/{AUTH-TOKEN} -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -H "Authorization: bearer {bearer_token}"

Proxy without username and password

curl -vvv --proxy http://{proxyIp}:{proxyPort} https://{apiServer}/api/v2//gateway/connectionNode/{AUTH-TOKEN} -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -H "Authorization: bearer {bearer_token}"

Proxy with username and password

curl -vvv --proxy http://{proxyIp}:{proxyPort}  --proxy-user {proxyUsername}:{proxyPasswd} https://{apiServer}/api/v2//gateway/connectionNode/{AUTH-TOKEN} -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -H "Authorization: bearer {bearer_token}"

Verify vProbe Container Connectivity.
- Login to vProbe Container:
```
kubectl exec -it nextgen-gw-0 -c vprobe -- bash
```
- Check OpsRamp Connection Grid IP connectivity:
```
telnet {csnode ip} 443
```
- Ensure OpenSSL works:
  - Direct Connection
```
openssl s_client -connect <csnode ip>:443 
```
  - Proxy Connection
```
openssl s_client -connect "csnode ip:443" -proxy 10.252.36.1:80
```
    Note: If we get 104 error, ask client if the ssl inspection is enabled. (SSL inspection is not enabled. Hence, client should disable it)
Whitelist OpsRamp IPs:
- Refer to OpsRamp Public IP Addresses document.
- Contact OpsRamp SaaS team if you don’t find the public APIs of the required POD.
Ensure SSL inspection is disabled for OpsRamp traffic.

Ensure there is no packet loss. To take the tcp dump / packet capture, run the following command:

tcpdump -v -i any 'ip host {proxy.ip or csnode ip} && tcp port {proxy.port or 443}' -s 3000 -w /tmp/{filename}.pcap

Collect the packet capture file and open it in Wireshark in your local machine and observe the packets as follows:
- Retransmission
- window size zero
- window size full

Gateway Tunnel Dropped Suddenly

Follow the below Troubleshooting steps, if Gateway tunnel is dropped suddenly and won’t reconnect.

Check vprobe-tls-comm.log for Exceptions:
- Open the vprobe-tls-comm.log file located in the /var/log/app/ directory.
- Look for the following exceptions and take necessary actions:
  - Connection reset
  - Connection timeout
  - No route to the host
  - Name resolution failed
  - Outbound closed
  - Read timeout
- Ensure that no new rules have been added to the firewall or network devices. OpsRamp traffic should be whitelisted, and SSL inspection for OpsRamp Gateway traffic should be disabled.
Verify DNS Resolution:
- Launch the debug container and run the following commands to check DNS resolution:
```
nslookup {api_server}
nslookup {csnode_server}
```
- Check the output and make sure if the OpsRamp servers are resolving correctly.
Check for Packet Loss or ISP Issues:
- Launch the debug container and run following command.
```
mtr {csnode}
mtr {api_server}
```
Monitor for High Latency:
- High latency within the customer’s infrastructure can cause issues. Run the following commands multiple times:
  - For Direct Connection:
```
netstat -anp | grep :443 | grep vprobe
```
  - For Proxy Connection:
```
netstat -anp | grep :<proxyport> | grep vprobe
```
- Observe the above command output. Check if you see any lag on Send-Q or Recv-Q.