Kali Linux

Kubeeye : Tool To Find Various Problems On Kubernetes

KubeEye is an inspection tool for Kubernetes. It discovers whether Kubernetes resources (by using OPA ), cluster components, cluster nodes (by using Node-Problem-Detector), and other configurations comply with best practices and makes modification suggestions accordingly.

KubeEye supports custom inspection rules and plugin installation. With KubeEye Operator, you can intuitively view the inspection results and modification suggestions on the web console.

Architecture

KubeEye obtains cluster resource details by using Kubernetes APIs, inspects resource configurations by using inspection rules and plugins, and generates inspection results. The architecture of KubeEye is as follows:

Install and use KubeEye

Install KubeEye on your machine.
- Method 1: Download the pre-built executable file from Releases.
- Method 2: Build from the source code.
Note: KubeEye files will be generated in /usr/local/bin/ on your machine. git clone https://github.com/kubesphere/kubeeye.git cd kubeeye make installke
(Optional) Install Node-problem-Detector. Note: If you need detailed reports, run the following command, and then NPD will be installed on your cluster. kubeeye install npd
Run KubeEye to inspect clusters.

Note: The results of KubeEye are sorted by resource kind.

kubeeye audit
KIND          NAMESPACE        NAME                                                           REASON                                        LEVEL    MESSAGE
Node                           docker-desktop                                                 kubelet has no sufficient memory available   warning    KubeletHasNoSufficientMemory
Node                           docker-desktop                                                 kubelet has no sufficient PID available      warning    KubeletHasNoSufficientPID
Node                           docker-desktop                                                 kubelet has disk pressure                    warning    KubeletHasDiskPressure
Deployment    default          testkubeeye                                                                                                                  NoCPULimits
Deployment    default          testkubeeye                                                                                                                  NoReadinessProbe
Deployment    default          testkubeeye                                                                                                                  NotRunAsNonRoot
Deployment    kube-system      coredns                                                                                                               NoCPULimits
Deployment    kube-system      coredns                                                                                                               ImagePullPolicyNotAlways
Deployment    kube-system      coredns                                                                                                               NotRunAsNonRoot
Deployment    kubeeye-system   kubeeye-controller-manager                                                                                            ImagePullPolicyNotAlways
Deployment    kubeeye-system   kubeeye-controller-manager                                                                                            NotRunAsNonRoot
DaemonSet     kube-system      kube-proxy                                                                                                            NoCPULimits
DaemonSet     k          ube-system      kube-proxy                                                                                                            NotRunAsNonRoot
Event         kube-system      coredns-558bd4d5db-c26j8.16d5fa3ddf56675f                      Unhealthy                                    warning   Readiness probe failed: Get "http://10.1.0.87:8181/ready": dial tcp 10.1.0.87:8181: connect: connection refused
Event         kube-system      coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9                      Unhealthy                                    warning   Readiness probe failed: HTTP probe failed with statuscode: 503
Event         kube-system      vpnkit-controller.16d5ac2b2b4fa1eb                             BackOff                                      warning   Back-off restarting failed container
Event         kube-system      vpnkit-controller.16d5fa44d0502641                             BackOff                                      warning   Back-off restarting failed container
Event         kubeeye-system   kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c   Failed                                       warning   Failed to pull image "controller:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Event         kubeeye-system   kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527   Failed                                       warning   Error: ImagePullBackOff
Role          kubeeye-system   kubeeye-leader-election-role                                                                                          CanDeleteResources
ClusterRole                    kubeeye-manager-role                                                                                                  CanDeleteResources
ClusterRole                    kubeeye-manager-role                                                                                                  CanModifyWorkloads
ClusterRole                    vpnkit-controller                                                                                                     CanImpersonateUser
ClusterRole                    vpnkit-controller                                                                                           CanDeleteResources

How KubeEye can help you

It inspects cluster resources according to Kubernetes best practices to ensure that clusters run stably.
It detects the control plane problems of the cluster, including kube-apiserver, kube-controller-manager, and etcd.
It detects node problems, including memory, CPU, disk pressure, and unexpected kernel error logs.

Checklist

Yes/No	Check Item	Description	Severity
✅	PrivilegeEscalationAllowed	Privilege escalation is allowed.	danger
✅	CanImpersonateUser	The Role/ClusterRole can impersonate users.	warning
✅	CanModifyResources	The Role/ClusterRole can delete Kubernetes resources.	warning
✅	CanModifyWorkloads	The Role/ClusterRole can modify Kubernetes resources.	warning
✅	NoCPULimits	No CPU limits are set.	danger
✅	NoCPURequests	No CPU resources are reserved.	danger
✅	HighRiskCapabilities	High-risk features, such as ALL, SYS_ADMIN, and NET_ADMIN, are enabled.	danger
✅	HostIPCAllowed	HostIPC is set to `true`.	danger
✅	HostNetworkAllowed	HostNetwork is set to `true`.	danger
✅	HostPIDAllowed	HostPID is set to `true`.	danger
✅	HostPortAllowed	HostPort is set to `true`.	danger
✅	ImagePullPolicyNotAlways	The image pull policy is not set to `always`.	warning
✅	ImageTagIsLatest	The image tag is `latest`.	warning
✅	ImageTagMiss	The image tag is missing.	danger
✅	InsecureCapabilities	Insecure options are missing, such as KILL, SYS_CHROOT, and CHOWN.	danger
✅	NoLivenessProbe	Liveless probe is not set.	warning
✅	NoMemoryLimits	No memory limits are set.	danger
✅	NoMemoryRequests	No memory resources are reserved.	danger
✅	NoPriorityClassName	Resource scheduling priority is not set.	ignore
✅	PrivilegedAllowed	Pods are running in the privileged mode.	danger
✅	NoReadinessProbe	Readiness probe is not set.	warning
✅	NotReadOnlyRootFilesystem	readOnlyRootFilesystem is not set to `true`.	warning
✅	NotRunAsNonRoot	runAsNonRoot is not set to `true`.	warning
✅	CertificateExpiredPeriod	The certificate expiry date of the API Server is less than 30 days.	danger
✅	EventAudit	Events need to be audited.	warning
✅	NodeStatus	Node status needs to be checked.	warning
✅	DockerStatus	Docker status needs to be checked.	warning
✅	KubeletStatus	kubelet status needs to be checked.	warning

Add your own inspection rules

Add custom OPA rules

Create a directory for storing OPA rules.

mkdir opa

Add custom OPA rule files.

Note:

OPA rule for checking workloads: The package name must be kubeeye_workloads_rego.
OPA rule for checking RBAC settings: The package name must be kubeeye_RBAC_rego.
OPA rule for checking node settings: The package name must be kubeeye_nodes_rego.

To check whether the image registry address complies with rules, save the following rules to imageRegistryRule.rego

package kubeeye_workloads_rego
deny[msg] {
    resource := input
    type := resource.Object.kind
    resourcename := resource.Object.metadata.name
    resourcenamespace := resource.Object.metadata.namespace
    workloadsType := {"Deployment","ReplicaSet","DaemonSet","StatefulSet","Job"}
    workloadsType[type]

    not workloadsImageRegistryRule(resource)

    msg := {
        "Name": sprintf("%v", [resourcename]),
        "Namespace": sprintf("%v", [resourcenamespace]),
        "Type": sprintf("%v", [type]),
        "Message": "ImageRegistryNotmyregistry"
    }
}

workloadsImageRegistryRule(resource) {
    regex.match("^myregistry.public.kubesphere/basic/.+", resource.Object.spec.template.spec.containers[_].image)
}

Run KubeEye with custom rules.

Note: Kubeeye will read all files ending with .rego in the directory.

root:# kubeeye audit -p ./opa
NAMESPACE     NAME              KIND          MESSAGE
default       nginx1            Deployment    [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot]
default       nginx11           Deployment    [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot]
default       nginx111          Deployment    [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]

Add custom NPD rules

Run the following command to change the ConfigMap: kubectl edit ConfigMap node-problem-detector-config -n kube-system
Run the following command to restart NPD: kubectl rollout restart DaemonSet node-problem-detector -n kube-system

KubeEye Operator

What is KubeEye Operator

KubeEye Operator is an inspection platform for Kubernetes. It manages KubeEye to regularly inspect clusters and generate inspection results.

How KubeEye Operator can help you

It records inspection results by using CR and provide a web page for you to intuitively view and compare cluster inspection results.
It provides more plugins.
It provides more detailed modification suggestions.

Deploy KubeEye Operator

kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml

Obtain the inspection results

kubectl get clusterinsight -o yaml

apiVersion: v1
items:
- apiVersion: kubeeye.kubesphere.io/v1alpha1
  kind: ClusterInsight
  metadata:
    name: clusterinsight-sample
    namespace: default
  spec:
    auditPeriod: 24h
  status:
    auditResults:
      auditResults:
      - resourcesType: Node
        resultInfos:
        - namespace: ""
          resourceInfos:
          - items:
            - level: warning
              message: KubeletHasNoSufficientMemory
              reason: kubelet has no sufficient memory available
            - level: warning
              message: KubeletHasNoSufficientPID
              reason: kubelet has no sufficient PID available
            - level: warning
              message: KubeletHasDiskPressure
              reason: kubelet has disk pressure
            name: kubeeyeNode

Click Here To Download

R K