KubeEye is an inspection tool for Kubernetes. It discovers whether Kubernetes resources (by using OPA ), cluster components, cluster nodes (by using Node-Problem-Detector), and other configurations comply with best practices and makes modification suggestions accordingly.
KubeEye supports custom inspection rules and plugin installation. With KubeEye Operator, you can intuitively view the inspection results and modification suggestions on the web console.
Architecture
KubeEye obtains cluster resource details by using Kubernetes APIs, inspects resource configurations by using inspection rules and plugins, and generates inspection results. The architecture of KubeEye is as follows:
Install and use KubeEye
- Install KubeEye on your machine.
- Method 1: Download the pre-built executable file from Releases.
- Method 2: Build from the source code.
/usr/local/bin/on your machine. git clone https://github.com/kubesphere/kubeeye.git cd kubeeye make installke - (Optional) Install Node-problem-Detector. Note: If you need detailed reports, run the following command, and then NPD will be installed on your cluster. kubeeye install npd
- Run KubeEye to inspect clusters.
Note: The results of KubeEye are sorted by resource kind.
kubeeye audit KIND NAMESPACE NAME REASON LEVEL MESSAGE Node docker-desktop kubelet has no sufficient memory available warning KubeletHasNoSufficientMemory Node docker-desktop kubelet has no sufficient PID available warning KubeletHasNoSufficientPID Node docker-desktop kubelet has disk pressure warning KubeletHasDiskPressure Deployment default testkubeeye NoCPULimits Deployment default testkubeeye NoReadinessProbe Deployment default testkubeeye NotRunAsNonRoot Deployment kube-system coredns NoCPULimits Deployment kube-system coredns ImagePullPolicyNotAlways Deployment kube-system coredns NotRunAsNonRoot Deployment kubeeye-system kubeeye-controller-manager ImagePullPolicyNotAlways Deployment kubeeye-system kubeeye-controller-manager NotRunAsNonRoot DaemonSet kube-system kube-proxy NoCPULimits DaemonSet k ube-system kube-proxy NotRunAsNonRoot Event kube-system coredns-558bd4d5db-c26j8.16d5fa3ddf56675f Unhealthy warning Readiness probe failed: Get "http://10.1.0.87:8181/ready": dial tcp 10.1.0.87:8181: connect: connection refused Event kube-system coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9 Unhealthy warning Readiness probe failed: HTTP probe failed with statuscode: 503 Event kube-system vpnkit-controller.16d5ac2b2b4fa1eb BackOff warning Back-off restarting failed container Event kube-system vpnkit-controller.16d5fa44d0502641 BackOff warning Back-off restarting failed container Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c Failed warning Failed to pull image "controller:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527 Failed warning Error: ImagePullBackOff Role kubeeye-system kubeeye-leader-election-role CanDeleteResources ClusterRole kubeeye-manager-role CanDeleteResources ClusterRole kubeeye-manager-role CanModifyWorkloads ClusterRole vpnkit-controller CanImpersonateUser ClusterRole vpnkit-controller CanDeleteResources
How KubeEye can help you
- It inspects cluster resources according to Kubernetes best practices to ensure that clusters run stably.
- It detects the control plane problems of the cluster, including kube-apiserver, kube-controller-manager, and etcd.
- It detects node problems, including memory, CPU, disk pressure, and unexpected kernel error logs.
Checklist
| Yes/No | Check Item | Description | Severity |
|---|---|---|---|
| ✅ | PrivilegeEscalationAllowed | Privilege escalation is allowed. | danger |
| ✅ | CanImpersonateUser | The Role/ClusterRole can impersonate users. | warning |
| ✅ | CanModifyResources | The Role/ClusterRole can delete Kubernetes resources. | warning |
| ✅ | CanModifyWorkloads | The Role/ClusterRole can modify Kubernetes resources. | warning |
| ✅ | NoCPULimits | No CPU limits are set. | danger |
| ✅ | NoCPURequests | No CPU resources are reserved. | danger |
| ✅ | HighRiskCapabilities | High-risk features, such as ALL, SYS_ADMIN, and NET_ADMIN, are enabled. | danger |
| ✅ | HostIPCAllowed | HostIPC is set to true. | danger |
| ✅ | HostNetworkAllowed | HostNetwork is set to true. | danger |
| ✅ | HostPIDAllowed | HostPID is set to true. | danger |
| ✅ | HostPortAllowed | HostPort is set to true. | danger |
| ✅ | ImagePullPolicyNotAlways | The image pull policy is not set to always. | warning |
| ✅ | ImageTagIsLatest | The image tag is latest. | warning |
| ✅ | ImageTagMiss | The image tag is missing. | danger |
| ✅ | InsecureCapabilities | Insecure options are missing, such as KILL, SYS_CHROOT, and CHOWN. | danger |
| ✅ | NoLivenessProbe | Liveless probe is not set. | warning |
| ✅ | NoMemoryLimits | No memory limits are set. | danger |
| ✅ | NoMemoryRequests | No memory resources are reserved. | danger |
| ✅ | NoPriorityClassName | Resource scheduling priority is not set. | ignore |
| ✅ | PrivilegedAllowed | Pods are running in the privileged mode. | danger |
| ✅ | NoReadinessProbe | Readiness probe is not set. | warning |
| ✅ | NotReadOnlyRootFilesystem | readOnlyRootFilesystem is not set to true. | warning |
| ✅ | NotRunAsNonRoot | runAsNonRoot is not set to true. | warning |
| ✅ | CertificateExpiredPeriod | The certificate expiry date of the API Server is less than 30 days. | danger |
| ✅ | EventAudit | Events need to be audited. | warning |
| ✅ | NodeStatus | Node status needs to be checked. | warning |
| ✅ | DockerStatus | Docker status needs to be checked. | warning |
| ✅ | KubeletStatus | kubelet status needs to be checked. | warning |
Add your own inspection rules
Add custom OPA rules
Create a directory for storing OPA rules.
mkdir opa
Add custom OPA rule files.
Note:
OPA rule for checking workloads: The package name must be kubeeye_workloads_rego.
OPA rule for checking RBAC settings: The package name must be kubeeye_RBAC_rego.
OPA rule for checking node settings: The package name must be kubeeye_nodes_rego.
To check whether the image registry address complies with rules, save the following rules to imageRegistryRule.rego
package kubeeye_workloads_rego
deny[msg] {
resource := input
type := resource.Object.kind
resourcename := resource.Object.metadata.name
resourcenamespace := resource.Object.metadata.namespace
workloadsType := {"Deployment","ReplicaSet","DaemonSet","StatefulSet","Job"}
workloadsType[type]
not workloadsImageRegistryRule(resource)
msg := {
"Name": sprintf("%v", [resourcename]),
"Namespace": sprintf("%v", [resourcenamespace]),
"Type": sprintf("%v", [type]),
"Message": "ImageRegistryNotmyregistry"
}
}
workloadsImageRegistryRule(resource) {
regex.match("^myregistry.public.kubesphere/basic/.+", resource.Object.spec.template.spec.containers[_].image)
}
- Run KubeEye with custom rules.
Note: Kubeeye will read all files ending with .rego in the directory.
root:# kubeeye audit -p ./opa NAMESPACE NAME KIND MESSAGE default nginx1 Deployment [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot] default nginx11 Deployment [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot] default nginx111 Deployment [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]
Add custom NPD rules
- Run the following command to change the ConfigMap: kubectl edit ConfigMap node-problem-detector-config -n kube-system
- Run the following command to restart NPD: kubectl rollout restart DaemonSet node-problem-detector -n kube-system
KubeEye Operator
What is KubeEye Operator
KubeEye Operator is an inspection platform for Kubernetes. It manages KubeEye to regularly inspect clusters and generate inspection results.
How KubeEye Operator can help you
- It records inspection results by using CR and provide a web page for you to intuitively view and compare cluster inspection results.
- It provides more plugins.
- It provides more detailed modification suggestions.
Deploy KubeEye Operator
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml
Obtain the inspection results
kubectl get clusterinsight -o yaml
apiVersion: v1
items:
- apiVersion: kubeeye.kubesphere.io/v1alpha1
kind: ClusterInsight
metadata:
name: clusterinsight-sample
namespace: default
spec:
auditPeriod: 24h
status:
auditResults:
auditResults:
- resourcesType: Node
resultInfos:
- namespace: ""
resourceInfos:
- items:
- level: warning
message: KubeletHasNoSufficientMemory
reason: kubelet has no sufficient memory available
- level: warning
message: KubeletHasNoSufficientPID
reason: kubelet has no sufficient PID available
- level: warning
message: KubeletHasDiskPressure
reason: kubelet has disk pressure
name: kubeeyeNode
, cluster components, cluster nodes){kind=link}