Dispatchers container causing my kamailio pod to CrashLoopBackOff in on-premise k8s cluster

[delliott3662]

Hi,

Thanks very much for your demo - it's been the only thing I've found that may address my SIP routing issues!

Though, I've been stuck on one problem for a week... I'm hoping someone could help me over this hurdle.
I'm running minikube on a Centos 7 VM.
Everything is Running with the exception of the kamailio pod, which is in CrashLoopBackOff.
Kubectl describe says the pod can't find /app:
exec: "/app": stat /app: no such file or directory: unknown

Any debug help would be much appreciated!
Thank you,
DRE

Commands and editing done:

$ minikube delete
$ minikube start
$ sudo swapoff -a
$ mkdir CyCore; cd CyCore
$ git clone https://github.com/CyCoreSystems/asterisk-k8s-demo.git
$ cd asterisk-k8s-demo/live-demo/k8s
$ vi 02-kamailio.yaml			// change CLOUD to "", change nodeSelector to "type: kamailio"
$ vi 03-asterisk.yaml			// change CLOUD to ""
$ kubectl label nodes minikube type=kamailio		// So kamailio pod will run on minikube node
$ kubectl apply -f 00-namespace.yaml			// so secret in voip will apply successfully
$ cd ../..
			// edited inbound.conf.tmpl, created config zip.
$ kubectl -n voip create secret generic asterisk-config --from-file=asterisk-config.zip
$ cd live-demo/k8s
$ kubectl apply -f 01-nats.yaml
$ kubectl apply -f 02-kamailio.yaml
$ kubectl apply -f 03-asterisk.yaml

Results:

$ kubectl get all --namespace=voip

NAME                            READY   STATUS             RESTARTS   AGE
pod/asterisk-758996b56d-t56c9   3/3     Running            4          95s
pod/kamailio-b96hs              2/3     CrashLoopBackOff   5          7m14s
pod/nats-666d56d6b8-q4bb5       1/1     Running            0          20m

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/asterisk   ClusterIP   10.105.56.62     <none>        5060/UDP                     20m
service/kamailio   ClusterIP   10.110.250.161   <none>        5060/UDP                     20m
service/nats       ClusterIP   10.104.131.110   <none>        4222/TCP,6222/TCP,8222/TCP   20m

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/kamailio   1         1         0       1            0           type=kamailio   20m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/asterisk   1/1     1            1           20m
deployment.apps/nats       1/1     1            1           20m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/asterisk-758996b56d   1         1         1       20m
replicaset.apps/nats-666d56d6b8       1         1         1       20m



$ kubectl describe pod kamailio-b96hs --namespace=voip 

Name:         kamailio-b96hs
Namespace:    voip
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Thu, 19 Aug 2021 08:56:05 -0400
Labels:       component=kamailio
              controller-revision-hash=5d4dd5d7cf
              pod-template-generation=3
Annotations:  <none>
Status:       Running
IP:           192.168.49.2
IPs:
  IP:           192.168.49.2
Controlled By:  DaemonSet/kamailio
Containers:
  kamailio:
    Container ID:   docker://90752fe442a294a2c37c44303d45dd6114dfe356663a40cfd7f403716331c7cc
    Image:          cycoresystems/kamailio
    Image ID:       docker-pullable://cycoresystems/kamailio@sha256:4a0804ab228a8eecd4abcf520a499af6119b4b30ae90a28abeec3f4008dc178b
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 19 Aug 2021 08:57:52 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      CLOUD:  
    Mounts:
      /data/kamailio from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-82wpn (ro)
  rtpproxy:
    Container ID:   docker://4a879cbfe529ee7c98efc452148b74caabeebbf6c93b8439839c4aa28cb9f69f
    Image:          cycoresystems/rtpproxy
    Image ID:       docker-pullable://cycoresystems/rtpproxy@sha256:e1443ca3b3a2f75cd0f26fb10e439cd6e920883d2ab1784962e56e2c01526beb
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 19 Aug 2021 08:58:08 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      CLOUD:  
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-82wpn (ro)
  dispatchers:
    Container ID:  docker://73389818732a94e43e3d8a62398b80f901aa6a1e8189569dc1f38a2e764a8ea4
    Image:         cycoresystems/dispatchers
    Image ID:      docker-pullable://cycoresystems/dispatchers@sha256:95f9cefb9d8454a907832f5d02d28a7f76f79e70c9a53c9c9a3aad4c700f27a0
    Port:          <none>
    Host Port:     <none>
    Command:
      /app
      -set
      voip:asterisk=1:5080
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/app": stat /app: no such file or directory: unknown
      Exit Code:    127
      Started:      Thu, 19 Aug 2021 09:01:29 -0400
      Finished:     Thu, 19 Aug 2021 09:01:29 -0400
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /data/kamailio from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-82wpn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-82wpn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-82wpn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  type=kamailio
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m25s                   default-scheduler  Successfully assigned voip/kamailio-b96hs to minikube
  Normal   Pulling    7m24s                   kubelet            Pulling image "cycoresystems/kamailio"
  Normal   Pulled     5m39s                   kubelet            Successfully pulled image "cycoresystems/kamailio" in 1m45.372032898s
  Normal   Created    5m38s                   kubelet            Created container kamailio
  Normal   Started    5m38s                   kubelet            Started container kamailio
  Normal   Pulling    5m38s                   kubelet            Pulling image "cycoresystems/rtpproxy"
  Normal   Pulled     5m22s                   kubelet            Successfully pulled image "cycoresystems/rtpproxy" in 16.111961198s
  Normal   Created    5m22s                   kubelet            Created container rtpproxy
  Normal   Started    5m22s                   kubelet            Started container rtpproxy
  Normal   Pulled     5m7s                    kubelet            Successfully pulled image "cycoresystems/dispatchers" in 15.121616868s
  Normal   Pulled     5m2s                    kubelet            Successfully pulled image "cycoresystems/dispatchers" in 3.614521805s
  Warning  Failed     4m44s (x3 over 5m6s)    kubelet            Error: failed to start container "dispatchers": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/app": stat /app: no such file or directory: unknown
  Normal   Pulled     4m44s                   kubelet            Successfully pulled image "cycoresystems/dispatchers" in 4.063114355s
  Normal   Pulling    4m21s (x4 over 5m22s)   kubelet            Pulling image "cycoresystems/dispatchers"
  Normal   Created    4m13s (x4 over 5m6s)    kubelet            Created container dispatchers
  Normal   Pulled     4m13s                   kubelet            Successfully pulled image "cycoresystems/dispatchers" in 8.398882695s
  Warning  BackOff    2m18s (x12 over 4m44s)  kubelet            Back-off restarting failed container

[sergeyu]

Just change the command path for 'dispatchers' daemon image in '02-kamailio.yaml' file to '/go/bin/dispatchers':

          image: cycoresystems/dispatchers
          command:
            - /go/bin/dispatchers
            - "-set"
            - voip:asterisk=1:5080

[delliott3662]

Thank you Sergei! You rock!! All pods are up and running!

I am seeing a couple errors in the container logs, wondering if you have any more magic bullets you can softly lob my way?

- kamailio/kamailio: "ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it"
- kamailio/dispatchers: "Failed to watch *v1.EndpointSlice"
- asterisk/asterisk: loading errors, but it comes up Ready.
- asterisk/natsgw: "failed to connect to Asterisk"

The only difference from the commands and editing that I enumerated above was your suggested change in 02-kamailio.yaml, and applying 00-rbac.yaml before applying 01-nats.yaml.

Appreciate your time,
DRE

$ kubectl get all --namespace=voip

NAME                            READY   STATUS    RESTARTS   AGE
pod/asterisk-758996b56d-pvbqh   3/3     Running   1          13m
pod/kamailio-5lklk              3/3     Running   3          13m
pod/nats-666d56d6b8-nt6nq       1/1     Running   0          13m

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/asterisk   ClusterIP   10.109.56.128   <none>        5060/UDP                     13m
service/kamailio   ClusterIP   10.102.58.75    <none>        5060/UDP                     13m
service/nats       ClusterIP   10.110.36.234   <none>        4222/TCP,6222/TCP,8222/TCP   13m

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/kamailio   1         1         1       1            1           type=kamailio   13m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/asterisk   1/1     1            1           13m
deployment.apps/nats       1/1     1            1           13m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/asterisk-758996b56d   1         1         1       13m
replicaset.apps/nats-666d56d6b8       1         1         1       13m


$ kubectl logs kamailio-5lklk --namespace=voip kamailio

 0(1) INFO: pv [pv_shv.c:60]: shvar_init_locks(): locks array size 16
 0(1) INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module
loading modules under config path: /usr/local/lib64/kamailio/modules:/usr/local/lib64/kamailio/modules_k:/usr/lib/x86_64-linux-gnu/kamailio/modules/
Listening on 
             udp: 192.168.49.2:5060 advertise 71.51.189.81:5060
             udp: 192.168.49.2:5080
Aliases: 
             udp: minikube:5080
             udp: minikube:5060
             *: 71--51-189-81.chvl.centurylink.net:*
             *: 71.51.189.81:*

 0(1) WARNING: <core> [core/daemonize.c:596]: mem_lock_pages(): failed to lock the memory pages (disable swap): Cannot allocate memory [12]
 0(1) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded
 0(1) INFO: rr [rr_mod.c:177]: mod_init(): outbound module not available
 0(1) INFO: permissions [permissions.c:598]: mod_init(): default allow file (/etc/kamailio/permissions.allow) not found => empty rule set
 0(1) INFO: permissions [permissions.c:612]: mod_init(): default deny file (/etc/kamailio/permissions.deny) not found => empty rule set
 0(1) INFO: permissions [trusted.c:180]: init_trusted(): db_url parameter of permissions module not set, disabling allow_trusted
 0(1) INFO: permissions [address.c:289]: init_addresses(): db_url parameter of permissions module not set, disabling allow_address
 0(1) INFO: utils [utils.c:255]: mod_init(): forward functionality disabled 0(1) INFO: utils [utils.c:177]: pres_db_init(): xcap_auth_status function is disabled
 0(1) INFO: <core> [main.c:2669]: main(): processes (at least): 22 - shm size: 67108864 - pkg size: 8388608
 0(1) INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
 0(1) INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 212992
 0(1) INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
 0(1) INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 212992
 4(29) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
 4(29) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
 4(29) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
 4(29) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
 1(26) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
 1(26) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
 1(26) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
 1(26) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
 3(28) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
 3(28) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
 3(28) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
 3(28) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
10(35) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
10(35) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
10(35) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
10(35) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
12(37) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
13(38) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
 5(30) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
 5(30) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
 5(30) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
 5(30) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
// repeat
18(43) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
18(43) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
18(43) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
18(43) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
17(42) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
17(42) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
17(42) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily

$ kubectl logs kamailio-5lklk --namespace=voip rtpproxy

INFO:main:GLOBAL: rtpproxy started, pid 1


$ kubectl logs kamailio-5lklk --namespace=voip dispatchers

// lots of these "Failed to watch *v1.EndpointSlice"
E0825 08:50:44.664650       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:voip:default" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
2021/08/25 08:50:58 current sets:
2021/08/25 08:50:58   set 1: []


$ kubectl logs asterisk-758996b56d-pvbqh --namespace=voip asterisk

Unable to set high priority
�[0mXSLT support not found. XML documentation may be incomplete.
XSLT support not found. XML documentation may be incomplete.
PBX UUID: 3c6b7ce2-dc6c-4c38-923a-f53f3c2130cc
Unable to load config file 'stasis.conf'
Could not load Stasis configuration; using defaults
[Aug 25 08:42:12] �[1;33mNOTICE�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2377�[0m �[1;37mload_modules�[0m: 113 modules will be loaded.
[Aug 25 08:42:12] �[1;31mWARNING�[0m[1]: �[1;37mccss.c�[0m:�[1;37m4386�[0m �[1;37minitialize_cc_max_requests�[0m: Could not find valid ccss.conf file. Using cc_max_requests default
[Aug 25 08:42:12] �[1;31mWARNING�[0m[1]: �[1;37mccss.c�[0m:�[1;37m4443�[0m �[1;37minitialize_cc_devstate_map�[0m: Could not find valid ccss.conf file. Using cc_[state]_devstate defaults
[Aug 25 08:42:12] �[1;33mNOTICE�[0m[1]: �[1;37mcdr.c�[0m:�[1;37m4517�[0m �[1;37mcdr_toggle_runtime_options�[0m: CDR simple logging enabled.
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mconfig_options.c�[0m:�[1;37m710�[0m �[1;37maco_process_config�[0m: Unable to load config file 'cel.conf'
[Aug 25 08:42:12] �[1;33mNOTICE�[0m[1]: �[1;37mcel.c�[0m:�[1;37m1636�[0m �[1;37mload_module�[0m: Failed to process CEL configuration; using defaults
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mconfig_options.c�[0m:�[1;37m710�[0m �[1;37maco_process_config�[0m: Unable to load config file 'features.conf'
[Aug 25 08:42:12] �[1;33mNOTICE�[0m[1]: �[1;37mfeatures_config.c�[0m:�[1;37m1874�[0m �[1;37mload_config�[0m: Could not load features config; using defaults
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mconfig_options.c�[0m:�[1;37m710�[0m �[1;37maco_process_config�[0m: Unable to load config file 'acl.conf'
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mconfig_options.c�[0m:�[1;37m710�[0m �[1;37maco_process_config�[0m: Unable to load config file 'udptl.conf'
[Aug 25 08:42:12] �[1;33mNOTICE�[0m[1]: �[1;37mudptl.c�[0m:�[1;37m1338�[0m �[1;37m__ast_udptl_reload�[0m: Could not load udptl config; using defaults
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m320�[0m �[1;37msorcery_config_internal_load�[0m: Unable to load config file 'pjproject.conf'
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:12] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig_options.c�[0m:�[1;37m710�[0m �[1;37maco_process_config�[0m: Unable to load config file 'pjsip_notify.conf'
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m1933�[0m �[1;37mprocess_text_line�[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mres_sorcery_config.c�[0m:�[1;37m326�[0m �[1;37msorcery_config_internal_load�[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] �[1;33mNOTICE�[0m[1]: �[1;37mconfbridge/conf_config_parser.c�[0m:�[1;37m2304�[0m �[1;37mverify_default_profiles�[0m: Adding default_bridge profile to app_confbridge
[Aug 25 08:42:13] �[1;33mNOTICE�[0m[1]: �[1;37mconfbridge/conf_config_parser.c�[0m:�[1;37m2315�[0m �[1;37mverify_default_profiles�[0m: Adding default_user profile to app_confbridge
[Aug 25 08:42:13] �[1;33mNOTICE�[0m[1]: �[1;37mconfbridge/conf_config_parser.c�[0m:�[1;37m2326�[0m �[1;37mverify_default_profiles�[0m: Adding default_menu menu to app_confbridge
[Aug 25 08:42:13] �[1;31mWARNING�[0m[1]: �[1;37mapp_voicemail.c�[0m:�[1;37m14597�[0m �[1;37mactual_load_config�[0m: Failed to load configuration file.
[Aug 25 08:42:13] �[1;31mWARNING�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2381�[0m �[1;37mload_modules�[0m: Some non-required modules failed to load.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2396�[0m �[1;37mload_modules�[0m: Error loading module 'res_pjsip_registrar_expire.so': /usr/lib/asterisk/modules/res_pjsip_registrar_expire.so: cannot open shared object file: No such file or directory
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2396�[0m �[1;37mload_modules�[0m: Failed to resolve dependencies for res_pjsip_transport_websocket
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2396�[0m �[1;37mload_modules�[0m: res_pjsip_transport_websocket declined to load.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2396�[0m �[1;37mload_modules�[0m: res_pjsip_notify declined to load.
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2396�[0m �[1;37mload_modules�[0m: Failed to resolve dependencies for res_ari_events
[Aug 25 08:42:13] �[1;31mERROR�[0m[1]: �[1;37mloader.c�[0m:�[1;37m2396�[0m �[1;37mload_modules�[0m: res_ari_events declined to load.
[Aug 25 08:42:13] �[1;31mWARNING�[0m[1]: �[1;37mconfig.c�[0m:�[1;37m2020�[0m �[1;37mprocess_text_line�[0m: No '=' (equal sign) in line 2 of /etc/asterisk/cli.conf
�[1;32mAsterisk Ready.�[0m


$ kubectl logs asterisk-758996b56d-pvbqh --namespace=voip config

2021/08/25 08:43:41 running service
2021/08/25 08:43:41 Waiting for Asterisk to be ready...
2021/08/25 08:43:49 change detected
2021/08/25 08:44:08 change detected


$ kubectl logs asterisk-758996b56d-pvbqh --namespace=voip natsgw

t=2021-08-25T08:42:11+0000 lvl=info msg="starting ari-proxy server" version=5.0.1
t=2021-08-25T08:42:11+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: dial tcp 127.0.0.1:8088: connect: connection refused"
t=2021-08-25T08:42:12+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: dial tcp 127.0.0.1:8088: connect: connection refused"
t=2021-08-25T08:42:13+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: bad status"
///these every second
t=2021-08-25T08:52:13+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: bad status"

[akasi88]

Hi,
I've been stuck on one problem for a month... I hope someone can help me overcome this obstacle.
I'm running k8s on a Centos 8 VM.
Everything is Running with the exception of the kamailio pod, which is in CrashLoopBackOff.

Results:

``[root@local live-demo]# kubectl get all --namespace=voip
NAME READY STATUS RESTARTS AGE
pod/asterisk-54d79f8659-gg5lp 3/3 Running 6 (9m7s ago) 155m
pod/kamailio-2lqwb 2/3 CrashLoopBackOff 6 (115s ago) 7m59s
pod/nats-d98bbdbf-ltng2 1/1 Running 0 22h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/asterisk ClusterIP 10.233.7.7 5060/UDP 22h
service/kamailio ClusterIP 10.233.20.49 5060/UDP 7m59s
service/nats ClusterIP 10.233.27.157 4222/TCP,6222/TCP,8222/TCP 22h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kamailio 1 1 0 1 0 kubernetes.io/hostname=node2 7m59s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/asterisk 1/1 1 1 155m
deployment.apps/nats 1/1 1 1 22h

NAME DESIRED CURRENT READY AGE
replicaset.apps/asterisk-54d79f8659 1 1 1 155m
replicaset.apps/nats-d98bbdbf 1 1 1 22h

[akasi88]

[root@local live-demo]# kubectl logs kamailio-2lqwb --namespace=voip kamailio
0(1) CRITICAL: [core/cfg.y:3507]: yyerror_at(): parse error in config file /data/kamailio/local.k, line 3, column 27-34: syntax error
0(1) CRITICAL: [core/cfg.y:3507]: yyerror_at(): parse error in config file /data/kamailio/local.k, line 3, column 27-34: hostname expected
0(1) CRITICAL: [core/cfg.y:3507]: yyerror_at(): parse error in config file /data/kamailio/local.k, line 4, column 1-6: syntax error
0(1) CRITICAL: [core/cfg.y:3507]: yyerror_at(): parse error in config file /data/kamailio/local.k, line 4, column 1-6:
ERROR: bad config file (4 errors)

[akasi88]

[root@local live-demo]# kubectl describe pod kamailio --namespace=voip
Name: kamailio-2lqwb
Namespace: voip
Priority: 0
Service Account: default
Node: node2/192.168.100.32
Start Time: Thu, 30 Nov 2023 17:37:08 +0500
Labels: component=kamailio
controller-revision-hash=565bc64649
pod-template-generation=1
Annotations:
Status: Running
IP: 192.168.100.32
IPs:
IP: 192.168.100.32
Controlled By: DaemonSet/kamailio
Containers:
kamailio:
Container ID: containerd://0dc218ca7b22ad3b8881562349ddd93124675849f8c03dadced5c00483171117
Image: cycoresystems/kamailio
Image ID: docker.io/cycoresystems/kamailio@sha256:4a0804ab228a8eecd4abcf520a499af6119b4b30ae90a28abeec3f4008dc178b
Port:
Host Port:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Thu, 30 Nov 2023 17:43:11 +0500
Finished: Thu, 30 Nov 2023 17:43:12 +0500
Ready: False
Restart Count: 6
Environment:
CLOUD:
Mounts:
/data/kamailio from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rtbr2 (ro)
rtpproxy:
Container ID: containerd://cfc9fae7fb59b680122019ff1ce8cab7be8e415142990df58dcf3a30b85e72ea
Image: cycoresystems/rtpproxy
Image ID: docker.io/cycoresystems/rtpproxy@sha256:e1443ca3b3a2f75cd0f26fb10e439cd6e920883d2ab1784962e56e2c01526beb
Port:
Host Port:
State: Running
Started: Thu, 30 Nov 2023 17:37:13 +0500
Ready: True
Restart Count: 0
Environment:
CLOUD:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rtbr2 (ro)
dispatchers:
Container ID: containerd://d408f5f157d2b5f35e533657a9490683588f4251e741c542424224d352c74965
Image: cycoresystems/dispatchers
Image ID: docker.io/cycoresystems/dispatchers@sha256:95f9cefb9d8454a907832f5d02d28a7f76f79e70c9a53c9c9a3aad4c700f27a0
Port:
Host Port:
Args:
-set
voip:asterisk=1:5080
State: Running
Started: Thu, 30 Nov 2023 17:37:15 +0500
Ready: True
Restart Count: 0
Environment:
Mounts:
/data/kamailio from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rtbr2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-rtbr2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=node2
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message

---

Normal Scheduled 10m default-scheduler Successfully assigned voip/kamailio-2lqwb to node2
Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/kamailio" in 2.025s (2.025s including waiting)
Normal Pulling 10m kubelet Pulling image "cycoresystems/rtpproxy"
Normal Created 10m kubelet Created container rtpproxy
Normal Pulling 10m kubelet Pulling image "cycoresystems/dispatchers"
Normal Started 10m kubelet Started container rtpproxy
Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/rtpproxy" in 1.797s (1.797s including waiting)
Normal Started 10m kubelet Started container dispatchers
Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/dispatchers" in 1.883s (1.883s including waiting)
Normal Created 10m kubelet Created container dispatchers
Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/kamailio" in 1.807s (1.807s including waiting)
Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/kamailio" in 1.828s (1.828s including waiting)
Normal Started 10m (x3 over 10m) kubelet Started container kamailio
Normal Created 10m (x3 over 10m) kubelet Created container kamailio
Normal Pulling 9m45s (x4 over 10m) kubelet Pulling image "cycoresystems/kamailio"
Warning BackOff 36s (x47 over 10m) kubelet Back-off restarting failed container kamailio in pod kamailio-2lqwb_voip(9a684224-d8dc-4562-8659-1ebe70e5fd3c)