Failure Modes and Processing in KIC

This reference describes the different types of Kong Ingress Controller failure modes and how it processes them.

When you run Kong Ingress Controller, you can encounter the following failures:

Kong Ingress Controller uses different methods to process each failure type, and creates error logs or other evidence, like Prometheus metrics and Kubernetes events, so you can observe and track the failures.

Errors in reconciling Kubernetes resources

When the controllers reconciling a specific kind of Kubernetes resource run into errors in reconciling the resource, a Reconciler error log line is recorded and the resource is re-queued for another round of reconciliation.

Thee Prometheus metric controller_runtime_reconcile_errors_total stores the total number of reconcile errors per controller from the start of Kong Ingress Controller. Search for the Reconciler error keyword in the Kong Ingress Controller container logs to see detailed errors.

Failures in translating configuration

When Kong Ingress Controller finds Kubernetes resources that can’t be correctly translated to Kong Gateway configuration (for example, an Ingress is using a non-existent Service as its backend), a translation failure is generated with the namespace and name of the objects causing the failure.

The Kubernetes objects causing translation failures are not translated to Kong Gateway configuration in the translation process. You can use Kubernetes events and Prometheus metrics to observe the translation failures. If Kong Ingress Controller is integrated with Kong Konnect, it will report that a translation error happened in the uploading node status.

Kong Ingress Controller collects all translation failures and generates a Kubernetes Event with the Warning type and the KongConfigurationTranslationFailed reason for each causing object in a translation failure. Prometheus metrics could also reflect the statistics of translation failures:

• ingress_controller_translation_broken_resource_count is the number of translation failures that happened in the latest translation
• ingress_controller_translation_count with the success=false label is the total number of translation procedures where translation failures happened

You can use kubectl get events -n <namespace> --field-selector reason="KongConfigurationTranslationFailed" to fetch events generated for translation failures. For example, if an Ingress named ing-1 in the namespace test used a non-existent Service as its backend, you could get the event with the following command:

kubectl get events -n test --field-selector reason="KongConfigurationTranslationFailed"

LAST SEEN   TYPE      REASON                               OBJECT                    MESSAGE
18m         Warning   KongConfigurationTranslationFailed   ingress/ing-1   failed to resolve Kubernetes Service for backend: failed to fetch Service test/httpbin-deployment-1: Service test/httpbin-deployment-1 not found

Failures in applying configuration to Kong Gateway

When Kong Ingress Controller fails to apply translated Kong Gateway configuration to Kong Gateway, Kong Ingress Controller will try to recover from the failure and record the failure into logs, Kubernetes events, and Prometheus metrics. Recovery usually fails because the translated configuration is rejected by Kong Gateway.

If Kong Ingress Controller fails to apply the translated configuration, it then tries to apply the last successful Kong Gateway configuration to new instances of Kong Gateway to attempt a best effort at making them available. If the FallbackConfiguration feature gate is enabled, Kong Ingress Controller discovers the Kubernetes objects that caused the invalid configuration, and tries to build a fallback configuration from valid objects and parts of the last valid configuration that are built from the broken objects. See fallback configuration for more information.

Debugging configuration failures

You can observe failures in applying configuration from Kubernetes events and Prometheus metrics:

• Kong Ingress Controller generates an event with the Warning type and the KongConfigurationApplyFailed reason attached to the pod itself when it fails to apply the configuration.
• For each object that causes the invalid configuration, Kong Ingress Controller generates a Warning event type and the KongConfigurationApplyFailed reason attached to the object.
• The Prometheus metric ingress_controller_configuration_push_count with the success=false label shows the total number of failures from applying the configuration by reason and URL of Kong Gateway admin API.
• The Prometheus metric ingress_controller_configuration_push_broken_resource_count reflects the number of Kubernetes resources that caused the error in the last configuration push.

If the CLI flag --dump-config is enabled, then the /debug/config/raw-error endpoint is enabled on the debug server port of Kong Ingress Controller, which will fetch the raw error returned from Kong Gateway if applying the configuration fails.

For example, let’s say you create an Ingress with the ImplementationSpecific path type and an invalid regex in Path (which can only be only be done when the validating webhook is disabled, otherwise it will be rejected by the webhook):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/strip-path: "true"
  name: ingress-invalid-regex
  namespace: default
spec:
  ingressClassName: kong
  rules:
  - http:
      paths:
      - backend:
          service:
            name: httpbin-deployment
            port:
              number: 80
        path: /~^^/a$
        pathType: ImplementationSpecific

You can get the Kubernetes events:

kubectl get events --all-namespaces --field-selector reason=KongConfigurationApplyFailed
NAMESPACE   LAST SEEN   TYPE      REASON                         OBJECT                                 MESSAGE
default     2m9s        Warning   KongConfigurationApplyFailed   ingress/ingress-invalid-regex          invalid paths.1: should start with: / (fixed path) or ~/ (regex path)
kong        15s         Warning   KongConfigurationApplyFailed   pod/kong-controller-779cb796f4-7q7c2   failed to apply Kong configuration to https://10.244.1.43:8444: HTTP status 400 (message: "failed posting new config to /config")

Both the events attached to the invalid ingress and attached to the Kong Ingress Controller pod are recorded.

Failures in uploading configuration to Konnect

When Kong Ingress Controller is integrated with Konnect and it fails to send configuration to Konnect, it generates error logs for failed requests, records the failures to Prometheus metrics, and updates the node status of itself in Konnect:

• Kong Ingress Controller parses errors returned from Konnect when uploading the configuration fails. It logs a line at the error level for each Kong Gateway entity that failed to create/update/delete, with the message Failed to send request to Konnect.
• The Prometheus metrics ingress_controller_configuration_push_count and ingress_controller_configuration_push_duration_milliseconds_bucket can also reflect configuration upload failures to Konnect, where the dataplane label is the URL of Konnect and success=false APIs.