Purpose

Going to show the common facing problem to check the connectivity between pods from different service points. If the service can not connect to the desired service via the service endpoint. Then what are the debugging steps we can follow to rectify that?

Background

We have two applications, each application running with two replica sets (i.e. 2 pods). The application names are app1 and app2.

Manifest for both apps are as given below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-deployment
  labels:
    app: app1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
      - name: app1
        image: nhossaincse/apiapp1:latest
        args:
          - --text=hello-world
        imagePullPolicy: Always
        ports:
        - containerPort: 80
app1-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app2-deployment
  labels:
    app: app2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app2
  template:
    metadata:
      labels:
        app: app2
    spec:
      containers:
      - name: app2
        image: nhossaincse/apiapp1:latest
        args:
          - --text=hello-world
        imagePullPolicy: Always
        ports:
        - containerPort: 8081
app2-deployment.yaml

We see that app1 running 80 post and the app2 running on 8081 port. We used a test image as the base of the apps, which is nhossaincse/apiapp1. apiapp1 is an in-memory web API, for a get request it serves the JSON. This is especially useful for demos or a more extensive "weatherforecast" Docker application. This is best for our case to simulate the scenario. As we are going to check the HTTP API call from app1 to app2.

Now we need connectivity of these two apps.

An example to simplify the objective is, we need to call an API of app2 from app1.  

To do this, first how app1 will reach to app2? This is elaboratively described in my earlier post which can be read from this link.

Kubernetes - Exposing Applications for Internal Access
Purpose Often we need to communicate between pods internally deployed in Kubernetes. As I said, internally means wanting to communicate without getting any public IP (which costs an extra $) rather in a local area network(LAN) manner. Background When pods are created, they are assigned an IP ad…
Nazmul HossainMd. Nazmul Hossain

So, as described in that post we are going to use the best to reach app2 from app1 with service.

Hence, we have now two apps and two services associated with each app. The service names are as app1-service and app2-service respectively.

Now the manifests for the two services are given below:

apiVersion: v1
kind: Service
metadata:
  name: app1-service
spec:
  selector:
    app: app1
  ports:
    - protocol: TCP
      port: 5678
      targetPort: 80
app1-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: app2-service
spec:
  selector:
    app: app2
  ports:
    - protocol: TCP
      port: 5678
      targetPort: 8081
app2-service.yaml

Problem statement (Issue or Challenge)

Now we will try to reach app2 from app1.

To do this first step is to go get into app1. We can do this by getting in an interactive session on app1. As app1 running with two pods, from anyone we can do the test. To get the list of pods below is the command:

$ kubectl get pods

The output of the running pods can be as below:

NAME                                 READY   STATUS    RESTARTS      AGE
app1-deployment-596b67c689-4m28f     1/1     Running   0             10m
app1-deployment-596b67c689-6qkqw     1/1     Running   0             10m
app2-deployment-64ddcf47dd-8qfh4     1/1     Running   0             10m
app2-deployment-64ddcf47dd-bb8dt     1/1     Running   0             10m

We can do this by getting in an interactive session on app1. The sample command to do this is as below:

$ kubectl exec -it <pod_name> /bin/bash

In our case, we want to do this for app1 pod so for the first pod to get into the interactive session:

$ kubectl exec -it app1-deployment-596b67c689-4m28f /bin/bash

Now we are in interactive mode inside the pod (think of this one as an independent Linux machine). Now we need to install some basic network packages inside the pod.

$ apt update -y && apt install net-tools -y && apt install dnsutils -y && apt install curl -y

Now to get  the app2 Fully Qualified Domain Name (FQDN), inside here we can use the nslookup command the argument with the <service_name>, in our case which is `app2-service`:

[ root@curl:/ ]$ nslookup app2-service
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      app2-service
Address 1: 10.106.29.144 app2-service.default.svc.cluster.local

We see that, the FQDN for the app2-service is app2-service.default.svc.cluster.local.

Now, we can do the curl to check the connectivity to app2-service.

[ root@curl:/ ]$ curl app2-service.default.svc.cluster.local:5679
curl: (7) Failed to connect to app2-service.default.svc.cluster.local port 5679: Connection refused

And, we see we got the error as port 5678: Connection refused.

Possible solutions

First, we need to check which port the port listening to. To do this we can run the below command to check which port actually the app2 pods listening to. In order to achieve this, we need to access app2 the pod as we did for app1 the pod with the below command in interactive mode. But, don't forget we need to install basic network utility as we did for app1 like running the command as $ apt update -y && apt install net-tools -y && apt install dnsutils -y && apt install curl -y.

Then, the below command will take us inside one of the pods of two with the pod name. Definitely, the pod name at your side will be different and you need to get that by running the command kubectl get pods.

$ kubectl exec -it app2-deployment-9598898-lm5sf /bin/bash

Now with the below command, we can check the ports that app2 listening to with the following command?

$ netstat -ntlp

The result would be like the below:

root@app2-deployment-9598898-lm5sf:/app# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::80                   :::*                    LISTEN      1/dotnet

We see that the port is listening to 80. So we need to correct in our manifest for the app2-deployment and app2-service.yaml with the port # 80

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app2-deployment
  labels:
    app: app2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app2
  template:
    metadata:
      labels:
        app: app2
    spec:
      containers:
      - name: app2
        image: hashicorp/http-echo
        args:
          - --text=hello-world
        imagePullPolicy: Always
        ports:
        - containerPort: 80
app2-deployment.yaml
apiVersion: v1
kind: Service
metadata:
  name: app2-service
spec:
  selector:
    app: app2
  ports:
    - protocol: TCP
      port: 5678
      targetPort: 80
app2-service.yaml

After that the curl to check the connectivity with the below command from the app1 pod as below:

root@app1-deployment-96b7f85bc-j5649:/app# curl app2-service.default.svc.cluster.local:5678/weatherforecast
[{"date":"2022-08-14T15:17:46.0946196+00:00","temperatureC":-2,"summary":"Freezing","temperatureF":29},{"date":"2022-08-15T15:17:46.095428+00:00","temperatureC":11,"summary":"Bracing","temperatureF":51},{"date":"2022-08-16T15:17:46.0954307+00:00","temperatureC":37,"summary":"Freezing","temperatureF":98},{"date":"2022-08-17T15:17:46.0954308+00:00","temperatureC":31,"summary":"Bracing","temperatureF":87},{"date":"2022-08-18T15:17:46.095431+00:00","temperatureC":53,"summary":"Sweltering","temperatureF":127}]root@app1-deployment-96b7f85bc-j5649:/app#

Finally, success! We can now connect the app2 form  app1. This process to connect rooms app2 to app1 will be the same as well. API call result with curl from app2 to app1 as below. 

root@app2-deployment-788d664db7-kst6d:/app# curl app1-service.default.svc.cluster.local:5678/weatherforecast
[{"date":"2022-08-14T15:23:39.454618+00:00","temperatureC":17,"summary":"Balmy","temperatureF":62},{"date":"2022-08-15T15:23:39.4554411+00:00","temperatureC":0,"summary":"Chilly","temperatureF":32},{"date":"2022-08-16T15:23:39.4554442+00:00","temperatureC":32,"summary":"Warm","temperatureF":89},{"date":"2022-08-17T15:23:39.4554444+00:00","temperatureC":20,"summary":"Hot","temperatureF":67},{"date":"2022-08-18T15:23:39.4554445+00:00","temperatureC":-15,"summary":"Sweltering","temperatureF":6}]root@app2-deployment-788d664db7-kst6d:/app#

Conclusion

We can use these steps to debug the connectivity problems between apps deployed in Kubernetes. This will help to detect the pain point in a quick manner.

Tagged in:

Kubernetes DevOps