解决Kubernetes 1.6.4 Dashboard无法访问的问题

前一段时间将之前采用kubeadm安装的Kubernetes 1.5.1环境升级到了1.6.4版本，升级过程较为顺利。由于该k8s cluster是一个测试环境，当时并没有过于关注，就忙别的事情了。最近项目组打算在这个环境下做一些事情，而当我们重新“捡起”这个环境时，发现Kubernetes Dashboard无法访问了。

Kubernetes的dashboard可以有很多种访问方式，比如：可以通过暴露nodeport的方式(无身份验证，不安全)、可以通过访问apiserver的api服务的方式等。我们的Dashboard通过APIServer进行访问：

http://apiserver_ip:secure_port/ui

正常情况下通过浏览器访问：http://apiserver_ip:secure_port/ui，浏览器会弹出身份验证对话框，待输入正确的用户名和密码后，便可成功进入Dashboard了。但当前，我们得到的结果却是：

User "system:anonymous" cannot proxy services in the namespace "kube-system".

而访问apiserver(http://apiserver_ip:secure_port/)得到的结果如下：

User "system:anonymous" cannot get  at the cluster scope.

一、问题原因分析

k8s 1.6.x版本与1.5.x版本的一个很大不同在于1.6.x版本启用了RBAC的Authorization mode(授权模型)，这点在K8s master init的日志中可以得到证实：

# kubeadm init --apiserver-advertise-address xx.xx.xx
... ...
[init] Using Kubernetes version: v1.6.4
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key
.... ...
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!
... ...

在《Kubernetes集群的安全配置》一文中我们提到过Kubernetes API server的访问方法：

Authentication(身份验证) -> Authorization（授权）-> Admission Control(入口条件控制)

只不过在Kubernetes 1.5.x及以前的版本中，Authorization的环节都采用了默认的配置，即”AlwaysAllow”，对访问APIServer并不产生什么影响：

# kube-apiserver -h
... ...
--authorization-mode="AlwaysAllow": Ordered list of plug-ins to do authorization on secure port. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC
... ...

但K8s 1.6.x版本中，–authorization-mode的值发生了变化：

# cat /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --allow-privileged=true
    ... ...
    - --basic-auth-file=/etc/kubernetes/basic_auth_file
    - --authorization-mode=RBAC
    ... ...

注：这里我们依旧通过basic auth方式进行apiserver的Authentication，而不是用客户端数字证书校验等其他方式。

显然问题的原因就在于这里RBAC授权方式的使用，让我们无法正常访问Dashboard了。

二、Kubernetes RBAC Authorization简介

RBAC Authorization的基本概念是Role和RoleBinding。Role是一些permission的集合；而RoleBinding则是将Role授权给某些User、某些Group或某些ServiceAccount。K8s官方博客《RBAC Support in Kubernetes》一文的中的配图对此做了很生动的诠释：

从上图中我们可以看到：

Role: pod-reader 拥有Pod的get和list permissions；
RoleBinding: pod-reader 将Role: pod-reader授权给右边的User、Group和ServiceAccount。

和Role和RoleBinding对应的是，K8s还有ClusterRole和ClusterRoleBinding的概念，它们不同之处在于：ClusterRole和ClusterRoleBinding是针对整个Cluster范围内有效的，无论用户或资源所在的namespace是什么；而Role和RoleBinding的作用范围是局限在某个k8s namespace中的。

Kubernetes 1.6.4安装时内建了许多Role/ClusterRole和RoleBinds/ClusterRoleBindings:

# kubectl get role -n kube-system
NAME                                        AGE
extension-apiserver-authentication-reader   50d
system:controller:bootstrap-signer          50d
system:controller:token-cleaner             50d

# kubectl get rolebinding -n kube-system
NAME                                 AGE
system:controller:bootstrap-signer   50d
system:controller:token-cleaner      50d

# kubectl get clusterrole
NAME                                           AGE
admin                                          50d
cluster-admin                                  50d
edit                                           50d
system:auth-delegator                          50d
system:basic-user                              50d
system:controller:attachdetach-controller      50d
... ...
system:discovery                               50d
system:heapster                                50d
system:kube-aggregator                         50d
system:kube-controller-manager                 50d
system:kube-dns                                50d
system:kube-scheduler                          50d
system:node                                    50d
system:node-bootstrapper                       50d
system:node-problem-detector                   50d
system:node-proxier                            50d
system:persistent-volume-provisioner           50d
view                                           50d
weave-net                                      50d

# kubectl get clusterrolebinding
NAME                                           AGE
cluster-admin                                  50d
kubeadm:kubelet-bootstrap                      50d
kubeadm:node-proxier                           50d
kubernetes-dashboard                           50d
system:basic-user                              50d
system:controller:attachdetach-controller      50d
... ...
system:controller:statefulset-controller       50d
system:controller:ttl-controller               50d
system:discovery                               50d
system:kube-controller-manager                 50d
system:kube-dns                                50d
system:kube-scheduler                          50d
system:node                                    50d
system:node-proxier                            50d
weave-net                                      50d

三、Dashboard的role和rolebinding

Kubernetes 1.6.x启用RBAC后，诸多周边插件也都推出了适合K8s 1.6.x的manifest描述文件，比如：weave-net等。Dashboard的manifest文件中也增加了关于rolebinding的描述，我当初用的是1.6.1版本，文件内容摘录如下：

// kubernetes-dashboard.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system
... ...

我们看到在kubernetes-dashboard.yaml中，描述文件新建了一个ClusterRoleBinding：kubernetes-dashboard。该binding将ClusterRole: cluster-admin授权给了一个ServiceAccount: kubernetes-dashboard。我们看看ClusterRole: cluster-admin都包含了哪些permission:

# kubectl get clusterrole/cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2017-05-30T14:06:39Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "11"
  selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolescluster-admin
  uid: 331c79dc-4541-11e7-bc9a-12584ec3a8c9
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

可以看到，在rules设定中，cluster-admin似乎拥有了“无限”权限。不过注意：这里仅仅授权给了一个service account，并没有授权给user或group。并且这里的kubernetes-dashboard是dashboard访问apiserver时使用的(下图右侧流程)，并不是user访问APIServer时使用的。

我们需要给登录dashboard或者说apiserver的user(图左侧)进行授权。

四、为user: admin进行授权

我们的kube-apiserver的启动参数中包含：

    - --basic-auth-file=/etc/kubernetes/basic_auth_file

也就是说我们访问apiserver使用的是basic auth的身份验证方式，而user恰为admin。而从本文开头的错误现象来看，admin这个user并未得到足够的授权。这里我们要做的就是给admin选择一个合适的clusterrole。但kubectl并不支持查看user的信息，初始的clusterrolebinding又那么多，一一查看十分麻烦。我们知道cluster-admin这个clusterrole是全权限的，我们就来将admin这个user与clusterrole: cluster-admin bind到一起：

# kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin
clusterrolebinding "login-on-dashboard-with-cluster-admin" created

# kubectl get clusterrolebinding/login-on-dashboard-with-cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: 2017-07-20T08:57:07Z
  name: login-on-dashboard-with-cluster-admin
  resourceVersion: "5363564"
  selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindingslogin-on-dashboard-with-cluster-admin
  uid: 686a3f36-6d29-11e7-8f69-00163e1001d7
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: admin

binding后，我们再来访问一下dashboard UI，不出意外的话，熟悉的dashboard界面就会出现在你的眼前。

用curl测试结果如下：

$curl -u admin:YOUR_PASSWORD -k http://apiserver_ip:secure_port/
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apps",
    "/apis/apps/v1beta1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/autoscaling/v2alpha1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v2alpha1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1beta1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1alpha1",
    "/apis/rbac.authorization.k8s.io/v1beta1",
    "/apis/settings.k8s.io",
    "/apis/settings.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/ping",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/ca-registration",
    "/healthz/poststarthook/extensions/third-party-resources",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/logs",
    "/metrics",
    "/swaggerapi/",
    "/ui/",
    "/version"
  ]
}