Kubernetes Dashboard. Installation Deep Dive

The deployment of applications and add-ons in Kubernetes are straightforward until those need to consume the Kubernetes API, that is the case of the Kubernetes Dashboard add-on. On version 1.7 of Kubernetes the RBAC service was introduced and many of those applications and add-ons started to crash.

This post will walk you through the process to deploy, configure and access to the Kubernetes Dashboard.

Kubernetes Dashboard Prerequisites

  • Running a Kubernetes platform 1.7.x and above.
  • Internet connection (pull Kubernetes Dashboard manifest and image)

If you don’t have a Kubernetes platform running at this time take a look to my post Hands-on Kubernetes: Deployment

Deploying Kubernetes Dashboard

On a node with kubectl command line installed run the following command. The manifest includes all the Kubernetes components to create for the add-on. Note: The K8s Dashboard URL has changed including the version, what it will make to fail the following command when new versions are released. If the following command doesn’t work, please make sure to visit the Kubernetes Dashboard repo and confirm the new URL.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

Check if your dashboard is running listing the pods in the namespace kube-system with the following command. You should see a kubernetes-dashboard-… pod with the status “running”.

kubectl -n kube-system get pod

Opening the dashboard

Access the dashboard at:

https://<master-ip>:<apiserver-port>/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

You likely got an error trying to access the dashboard.

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"system:anonymous\" cannot get services/proxy in the namespace \"kube-system\"",
  "reason": "Forbidden",
  "details": {
    "name": "https:kubernetes-dashboard:",
    "kind": "services"
  },
  "code": 403
}

At this point you will start to look for a solution on Internet. The solutions you mostly will find are the ones below, kubectl proxy and NodePort, but they are not recommended for production.

kubectl proxy

This access mode is not recommended to be used as the method to publicly expose your dashboard. The proxy only allows HTTP connection.

To use this method you need to install kubectl in your computer and run the following command. The proxy will start to serve the dashboard on http://localhost:8001 by default.

kubectl proxy

Personally I don’t recommend to use this connection method. If you are sharing a jump server or even on your own computer, a sniffer will able to capture your kubeconfig file or token since they are sent as plain text via HTTP.

You can find more information Accessing Dashboard 1.7.X and above.

NodePort

If you are running a single node setup (unlikely in production), you can configure the Kubernetes Dashboard service to use NodePort as the type to publish the service.

I’m not going to explain how to set the service type since the Kubernetes Dashboard site has a clear procedure (Accessing Dashboard 1.7.X and above)

API Server

This is the method which I recommend to use for production systems as well as for dev and test. It is important to keep the same security mechanisms end to end and get familiar with Kubernetes RBAC.

To use the API server you need to install the user certificates in the browser. I’m going to use the kubeconfig file generated by kubeadm, I want to keep this post as short as I can.

Tip: For production systems each user should have its own certificates. Bitnami have a great doc about how to configure it (Create User With Limited Namespace Access)

Let’s see how we can extract the certificates from the kubeconfig file:

  1. Locate your kubeconfig or config file which you use to run kubectl commands. If you have used my Vagrant file above, you can find it on /home/vagrant/.kube/config or /etc/kubernetes/admin.conf
  2. You need to export a single file (.p12) with the following two certificates: the client-certificate-data, and the client-key-data. My example runs the command on /home/vagrant. If you run this command on macOS, be sure to change the base64 -d to base64 -D.
    grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
    grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
    openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"
  3. Import the kubecfg.p12 certificate, reopen your browser, and visit the Kubernetes Dashboard URL. Accept any warning and you should see the authentication page. You can skip the login and check you are not able to perform any task.
  4. The following steps have been copied from the Kubernetes Dashboard wiki page (Creating-sample-user)
    1. Create service account
      cat <<EOF | kubectl create -f -
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: admin-user
        namespace: kube-system
      EOF
    2. Create ClusterRoleBinding
      cat <<EOF | kubectl create -f -
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: admin-user
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: cluster-admin
      subjects:
      - kind: ServiceAccount
        name: admin-user
        namespace: kube-system
      EOF
    3. Get the Bearer Token. Once you run the following command, copy the token value which you will use on the following step.
      kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
    4. Come back to your browser and choose token on the login page. You will need to paste the token value you have copied on the previous step.
    5. Click “SIGN IN” and you should be able to see your Kubernetes Dashboard fully operational.

Summary

API Server should be your choice when production systems. If you want your users to have each one their own certificates, which I encourage you to do, don’t miss the Bitnami post mentioned above.

Note: You will find on GitHub and other blogs the option to give cluster-admin access to system:anonymous. This is an easy way to not export certificates and create a cluster-admin service account. I highly discourage you to use this approach on any enterprise environment.

45 thoughts on “Kubernetes Dashboard. Installation Deep Dive”

  1. Hello,
    Thanks a lot this posts.
    I have one issue if you can help me with,

    while accessing the dashboard…

    Error: ‘Service Unavailable’
    Trying to reach: ‘https://10.32.0.5:8443/’

    My dashboard is unable to access the endpoint. Any ideas ?

      1. Hello,

        The master and the node, and also i’m behind proxy. i added no_proxy(ip_master, ip_node) variable before setting up my matser.

        Thanks a lot.

        1. Jose Luis Gomez

          Hi,

          I had lot of challenges when proxies are in the picture. You must add any range you are going to communicate to in the “no_proxy” section. Add there any overlay network as well as 127.0.0.1, localhost. Also if you are using FQDN, I recommend to add into your no_proxy the “.domain.local”

          Regards,
          Jose

  2. Hi,

    I’d like to know “Import the kubecfg.p12 certificate”. Where should the kubecfg.p12 be imported? Kubernetes master? or client server? and how to import the kubcfg.p12?

    Thanks a lot

    – Marslo

    1. Jose Luis Gomez

      Hi Marslo,

      Like is mentioned in the post, you need to import (install) the certificate on your browser.

      Regards,
      Jose

      1. Thanks Jose for your reply. I’m still confused with Kubeconfig way to login (token works for me).

        Here my situation:
        Kubernetes master: ubuntu 17.10; kubernetes version: v1.10.1.

        According to step2:
        1. generate kubecfg.crt
        2. generate kubecfg.key
        3. generate kubecfg.p12 by kubecfg.crt+kubecfg.key

        I’ve done all them. here three files: kubecfg.crt, kubecfg.key and kubecfg.p12.

        The kubecfg.crt has been imported by using:
        “`
        $ sudo cp kubecfg.crt /usr/local/share/ca-certificates/
        $ sudo update-ca-certificates
        Updating certificates in /etc/ssl/certs…
        1 added, 0 removed; done.
        Running hooks in /etc/ca-certificates/update.d…
        done.
        “`

        And I’ve also restarted the browser, and using kubecfg.p12 as the Kubeconfig file, but error shows:
        “`
        yaml: control characters are not allow
        “`

        So, I want to know how to import the p12 for Kubeconfig way to login?

        1. Jose Luis Gomez

          Hi Marslo,

          The .p12 is the certificate to install on your browser. This is not the token, this is the certificate to have installed on your Chrome, Firefox or whatever web browser you use.

          For the token you must continue with the step 4.

          Regards,
          Jose

  3. Hi Jose
    nice article. I am trying to use API server to get access to dashboard and completed steps #1 and #2, but still getting error in browser

    {
    “kind”: “Status”,
    “apiVersion”: “v1”,
    “metadata”: {},
    “status”: “Failure”,
    “message”: “services \”https:kubernetes-dashboard\” is forbidden: User \”system:anonymous\” cannot get services in the namespace \”kube-system\””,
    “reason”: “Forbidden”,
    “details”: {
    “name”: “https:kubernetes-dashboard”,
    “kind”: “services”
    },
    “code”: 403
    }

    I have also created the serviceaccount and ClusterRoleBinding

    1. Jose Luis Gomez

      Hi Rajan,

      You need to complete all the steps. You will continue seeing that message until you have installed the certificate on your web browser.

      Regards,
      Jose

  4. Venkat Ramani

    I have installed the certificate and I am able to look at the api server service list but when accessing the dashboard getting this error below. Been researching for hours to no avail .Appreciate any help.

    Error: ‘dial tcp 10.32.0.79:8443: getsockopt: no route to host’
    Trying to reach: ‘https://10.32.0.79:8443/’

  5. Thanks Jose, this is nice article 🙂

    I suffer a lot on the dashboard installation and access, I queried quite lot of articles, none of them works in the end. this article help me a lot!!

    Besides this I got a strange issue when “SIGN IN” dashboard, the chrome shows “Error: ‘dial tcp 10.244.1.8:8443: getsockopt: connection timed out'”

    To fix this, we need Add route at kube-master node:

    from to corresponding , it seems need be done whenever kubernetes-dashboard’s POD ip is changed

    Here is an example:

    $ sudo route add -net 10.244.1.12 netmask 255.255.255.255 gw 192.168.56.102
    $ sudo route add -net 10.244.2.10 netmask 255.255.255.255 gw 192.168.56.103

    1. Error: ‘Internal Privoxy Error’
      Trying to reach: ‘https://10.244.1.2:9090/’

      How about this error?

      As you see, 10.244.1.2 is thy podID of kubernetes dashboard.

      Do i need to add route at my master node?

      Thanks.

    2. xiaogauzh, i had similar situation and what I decided after doing some research to manually restart docker and kubelet service keeping everything as it is and that solved the problem

  6. Nice post.

    It helped me a lot.

    For Chrome browser, you can install .p12 certificate as below:

    Open browser, click Settings -> Advanced -> Privacy and security, and then open Manage certificates.

    On “Personal” tab, click Import, and choose file type as “All files”, and then choose downloaded kubecfg.p12 file, and input previous export password.

  7. Hello,

    Its a simply a great article, really helped me in moving forward. I spent days go through articles but nothing constructive was found.
    I am stuck at 2 points:
    1. Kubernetes dashboard is appearing in IE only. In Firefox and Chrome getting same error (Forbidden : 403)
    2. In IE even after submitting token, and clicking on “SIGN IN” nothing is happening.

    Please provide some pointers.

    1. Jose Luis Gomez

      Hi Nikhil,

      Confirm the certificate has been successfully installed on Chrome or Firefox. You can see on the post comments some guidance to do the installation on those browsers.

      About the token, the same. Have a look to the comments as well as the article to confirm you are using the right token.

      Regards,
      Jose

  8. Thanks for the tutorial. It works :). Now I started from scratch (I’m using kubernetes locally atm) and what I don’t understand is the following: I’ve installed an nginx pod with an ingress (tls) to the dasbhoard which works fine but now I can access the dashboard immediately without providing any token or config file. Any idea why this is happening?

    1. Jose Luis Gomez

      Hi,

      TBH, no idea. I would check accessing directly to the dashboard skipping the ingress controller and see if you get the auth page. If you don’t, you may have create some clusterolebinding during your ingress deployment that breaks the authentication.

      I’ll test this scenario later this week and see if I get the same behaviour.

      Regards,
      Jose

  9. Hi Jose Luis Gomez,

    I have a k8s cluster with single master node and single worker node on two different servers(coreos installed using PXE).

    ApiServer is running on insecure connections without any certificates. (http://:8080/version is working fine and also able to deploy sample pods eg. ngnix)

    Now I’m trying to setup dashboard with insecure connection (on port 80) but when run the command kubectl -f create YAMLFILE, dashboard docker container on worker node is failed to run with below error.

    [email protected] ~ $ docker logs 3e6633512633
    2018/08/05 17:41:41 Starting overwatch
    2018/08/05 17:41:41 Using apiserver-host location: http://:8080
    2018/08/05 17:41:41 Skipping in-cluster config
    2018/08/05 17:41:41 Using random key for csrf signing
    2018/08/05 17:41:41 No request provided. Skipping authorization
    2018/08/05 17:41:41 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the –apiserver-host param points to a server that does not exist. Reason: Get http://:8080/version: dial tcp :8080: getsockopt: no route to host
    Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

    Please help me getting the basic dashboard up and running. Later I’ll try with secured connections.

    1. Jose Luis Gomez

      Hi,

      Have a look to your YAML file and be sure to set the value for:

      Uncomment the following line to manually specify Kubernetes API server Host
      # If not specified, Dashboard will attempt to auto discover the API server and connect
      # to it. Uncomment only if the default does not work.
      # - --apiserver-host=http://my-address:port

      Regards,
      Jose

  10. Currently running a multi-node setup and am using NodePort to publish my dashboard service to the LAN. I can get to the dashboard and sign-in using my service account token just fine.

    Is there any reason I should not use NodePort for multi-node configs? It seems like you (and k8s) shy away from using NodePort in multi-node setups. Why is this?

    1. The reason of not use NodePort in a multi-node environment is because the K8s dashboard runs as a single replica. In the event of the pod crashes or the node restart, the K8s dashboard pod will we initiated on a different node and you will need to check which one and its IP address. The K8s dashboard IP will be dynamic and that creates operational challenges.

  11. Hi,

    Great article! I used it for setting up admin access to the dashbord, and it’s work great.

    Now it’s time to set up a user with access only to a single namespace (by the way, it is possible to have access to multiple namespaces? How?):

    apiVersion: v1
    kind: Namespace
    metadata:
    name: %NAMESPACE%

    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: %USER%-serviceaccount
    namespace: %NAMESPACE%

    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: Role
    metadata:
    name: %USER%-role
    namespace: %NAMESPACE%
    rules:
    – apiGroups: [“”, “extensions”, “apps”, “batch”]
    resources: [“*”]
    verbs: [“*”]

    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: %USER%-serviceaccount-binding
    namespace: %NAMESPACE%
    roleRef:
    kind: Role
    name: %USER%-role
    apiGroup: rbac.authorization.k8s.io
    subjects:
    – kind: ServiceAccount
    name: %USER%-serviceaccount
    namespace: %NAMESPACE%

    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
    name: %USER%-role-binding
    namespace: %NAMESPACE%
    roleRef:
    kind: Role
    name: %USER%-role
    apiGroup: “rbac.authorization.k8s.io”
    subjects:
    – kind: User
    name: %USER%
    apiGroup: “rbac.authorization.k8s.io”

    Creating cartificate and .p12 with the same command as you, and creating .kube/config. Now the user is able to use kubectl from his terminal. But he cannot access the dashboard (imported .p12 of course). Getting this error:

    services “https:kubernetes-dashboard:” is forbidden: User “system:%USER%” cannot get resource “services/proxy” in API group “” in the namespace “kube-system”

    I understand why getting this error: The %USER%-role is walid for his namespace only and not for kube-system.

    How can I resolve this?

    Thank you,

  12. Aman Shrivastava

    When you talk about import the certificate (.p12) and go to Kubernetes Dashboard URL, did you mean “https://:8443/”.

    I am not able to reach dashboard url. I am using 2 masters, along with external load balancer (HAProxy Load balancer).

    Can you help me with the url?

    Thanks in advance.

  13. Hi Jose

    I still get the same forbidden error even after completing all the steps that you mentioned.

    I imported the .p12 file into chrome and it prompted me for the private key password as well. And i also get the message that the import is successful.

    I also have the token to be fed into the pre signin page but unfortunately i get the forbidden error.

    Thanks
    Manikandan

    1. Hi Manikandan,

      I recommend you to try with different browsers. I have seen sometimes with Chrome that the certificates are not picked for some reason.

      Regards,
      Jose Gomez

  14. hi this is ram,
    iam getting issue while sigining dashboard
    Error: ‘dial tcp 10.244.2.2:8443: i/o timeout’
    Trying to reach: ‘https://10.244.2.2:8443/’

    help me

  15. Hi Jose,
    Your post is excellent. I was able to install certificates correctly, and am able to get response from all URLs like
    https://52.88.10.115:6443/
    https://52.88.10.115:6443/api, etc.

    However the dashboard URL (https://52.88.10.115:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/) gives error

    Error: ‘dial tcp 10.244.1.2:8443: i/o timeout’
    Trying to reach: ‘https://10.244.1.2:8443/’

    I have deployed the kubernetes cluster on AWS EC2 instances.

    Adding routes as suggested by some comments like

    sudo route add -net 10.244.1.2 netmask 255.255.255.255 gw 172.30.0.102

    is also not helping, though I am not sure if I am using the correct IPs.

    172.30.0.102 in the above snippet is the private IP of my worker node.
    10.244.1.2 is the IP provided in the error

    Where I am going wrong. I have been kind of struggling with this since quite some time. I will be grateful if someone can help out here.

    Thanks
    Parag

  16. I’ve run the steps as above, but the display still does not appear?

    message error:

    configmaps is forbidden: User “system:anonymous” cannot list resource “configmaps” in API group “” in the namespace “default”

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.