Sleep Mode + Auto-Delete

The sleep mode feature in Loft puts spaces or virtual clusters to sleep - either after a specified period of inactivity or manually triggered by a user. Spaces and virtual clusters automatically wake up when a user runs the first kubectl, helm or any other command using the space or virtual clusters. Users can also manually trigger the wake up process by clicking a button in the UI. Instead of the sleep mode, you are also able to configure automatic deletion of spaces.

Virtual Clusters

Technically, virtual clusters do not have a sleep mode on their own. Instead, you can enable the sleep mode for the underlying namespace that the virtual cluster runs in. For convenience and easier understanding, the Loft UI shows the sleep mode statistics for each virtual cluster but in fact the numbers reflect the sleep mode of the underlying space of each virtual cluster.

State Preservation

Note that when a space or virtual cluster goes to sleep, all persistent data as well as all Kubernetes objects will be preserved. Only the running containers will be terminated when the sleep mode starts. And these containers will be restarted once the space or virtual cluster wakes up again. Learn more about how sleep mode works.

Use Case & Benefits

While some of the spaces and virtual clusters that you manage with Loft may contain production workloads which must run all day every day, there are many use cases where you want to create spaces and virtual clusters for development, testing, experimentation or CI/CD. These non-production workloads are usually just required to run when engineers are actually working with them. Considering a regular 40-hour work week, engineers work about 24% of the overall time per week:

40h work time / (24h per day * 7 days) total hours per week) ≈ 24%

And this calculation is not even including time for meetings, sick days, holidays etc. That means you can save more than 76% of your cloud infrastructure cost for idle namespaces or virtual clusters. Some namespaces and virtual clusters may even be running without any engineer touching them for weeks because one of the biggest downsides of the cloud is that while it is easy to spin up things, hardly anyone ever shuts down anything without being forced to do so.

Sleep mode offers a fully automated solution for this problem. If your cluster is configured to horizontally auto-scale its nodes (which most public cloud providers allow you to enable with just a single click), you can save over 76% of infrastructure cost for your clusters by enabling sleep mode.

Sleep

When putting a space or virtual cluster to sleep, Loft looks up and saves the replica number for Deployments, StatefulSets, DaemonSets and other ReplicaSet based resources. Then, it scales down this replica number to 0 which triggers Kubernetes to delete all pods/containers.

With Loft, you can:

• enforce automatic sleep for all spaces of a certain cluster account after a certain period of inactivity
• enable automatic sleep for individual spaces after a certain period of inactivity
• trigger sleep mode manually for individual spaces

Automatic For Entire Account

Cluster admins can enforce the sleep mode annotation for all spaces of a certain account to ensure that all spaces go to sleep after a certain period of inactivity.

Enforcement

Annotations set in the account settings cannot be overridden by the user when creating a space. The default RBAC set up by Loft does not allow users to modify their namespaces after creation. If you add additional RBAC to allow users to update their namespaces, users will be able to deactivate or manually trigger sleep mode.

UI

Loft UI - Change Sleep Mode In Space Creation Settings

kubectl

Create file account.yaml:

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account
spec:
  subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: john          # References the `spec.subject` defined in the User object
  space:
    clusterRole: loft-cluster-space-default
    limit: 5
    spaceTemplate:
      metadata:
        creationTimestamp: null
        labels:
          some-label: some-value
        annotations:
          # start sleep mode after 1 hour (= 3600 seconds) of inactivity
          sleepmode.loft.sh/sleep-after: '3600'

Create/Update the account using kubectl:

# IMPORTANT: Make sure to switch to the context of the connected cluster!
kubectl apply -f account.yaml

Automatic For Individual Spaces

UI

Loft UI - Change Sleep Mode For Space

kubectl

Create file space.yaml:

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: ml-experiment-1
spec:
  account: janes-account
  annotations:
    # start sleep mode after 1 hour (= 3600 seconds) of inactivity
    sleepmode.loft.sh/sleep-after: '3600'

Create/Update the space using kubectl:

# IMPORTANT: Make sure to switch to the context of the connected cluster!
kubectl apply -f space.yaml

User Permissions

The default RBAC rules set up by Loft will not permit non-admin users to edit spaces/namespaces.

Manual Sleep

UI

Loft UI - Manual Sleep & Wake-Up

CLI

Run this command to manually put a space to sleep:

loft sleep [SPACE_NAME]   # optional flag: --cluster=[CLUSTER_NAME]

User Permissions

The default RBAC rules set up by Loft will not permit non-admin users to edit spaces/namespaces.

kubectl

Create file space.yaml:

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: ml-experiment-1
spec:
  account: johns-account
  annotations:
    # force this space to sleep immediately
    sleepmode.loft.sh/force: 'true'

Create/Update the space using kubectl:

# IMPORTANT: Make sure to switch to the context of the connected cluster!
kubectl apply -f space.yaml

User Permissions

The default RBAC rules set up by Loft will not permit non-admin users to edit spaces/namespaces.

Wake-Up

When Loft wakes up a space, it starts all pods/containers it has previously removed from the namespace. Loft does this by changing the replica number for Deployments, StatefulSets, DaemonSets and other ReplicaSet based resources.

Automatic Wake-Up

Loft is configured to wake up spaces automatically once they are being used. Using a space means to send any request to the Kubernetes API server involving any resources within the corresponding namespace. It does not matter if this request comes from running a kubectl command, a helm command or using any other tool, e.g.:

kubectl get po -n [NAMESPACE]

Manual Wake-Up

UI

Loft UI - Manual Sleep & Wake-Up

CLI

You can always wake up a space by running a kubectl command:

# IMPORTANT: Make sure you are in the context of this space (using the correct namespace)
kubectl get po

Instead of running a kubectl command, you can also run this command to manually wake up a space:

loft wakeup [SPACE_NAME]   # optional flag: --cluster=[CLUSTER_NAME]

User Permissions

The default RBAC rules set up by Loft will not permit non-admin users to edit spaces/namespaces.

kubectl

Create file space.yaml:

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: ml-experiment-1
  annotations:
    # remove force sleep annotation or set to false
    # sleepmode.loft.sh/force: 'false'
spec:
  account: johns-account

Create the account using kubectl:

# IMPORTANT: Make sure to switch to the context of the connected cluster!
kubectl apply -f space.yaml

User Permissions

The default RBAC rules set up by Loft will not permit non-admin users to edit spaces/namespaces.

How does it work?

Sleep mode is configured by annotations in the metadata of a Kubernetes namespace. After a certain amount of inactivity, the space starts sleeping.

How is the inactivity determined?

All requests that are made through the Loft API server count as activity in the namespace. Let's say you have created a new space via:

loft create space sleep-test

and then you access resources within the namespace that will count as activity in the namespace. Some examples:

# This counts as activity
kubectl --context loft_sleep-test_local get pods -n sleep-test

# This does NOT count as activity
kubectl --context non_loft_context get pods -n sleep-test

# This counts as activity
curl -k -v -XGET 'https://my.loft.instace.com/kubernetes/cluster/local/api/v1/namespaces/sleep-test/pods?limit=500'

# This does NOT count as activity (because of the added header X-Sleep-Mode-Ignore)
curl -k -v -XGET -H "X-Sleep-Mode-Ignore: true" 'https://my.loft.instace.com/kubernetes/cluster/local/api/v1/namespaces/sleep-test/pods?limit=500'

As you can see in the examples above, all kubernetes requests that are routed through the Loft API server are counted as activity within the namespace and will reset the period until it sleeps or is deleted. One exception to this rule are requests that are made in the Loft UI, since the Loft UI always sets the X-Sleep-Mode-Ignore: true header for each request.

There are multiple annotations you can set on a namespace to change what counts as activity within that namespace:

• sleepmode.loft.sh/last-activity: this is usually set automatically by Loft after an activity was detected within a namespace, however you can also change this value as you like
• sleepmode.loft.sh/ignore-all: if this annotation is set to "true", then all requests will be ignored and not count as activity
• sleepmode.loft.sh/ignore-vclusters: if "true", activity that occurs within a created vcluster within this namespace does not count as activity
• sleepmode.loft.sh/ignore-groups: a comma separated list of user groups that do not count as activity (e.g. loft:team:admins would ignore all users in the team admins or loft:user:admin would ignore the admin user)
• sleepmode.loft.sh/ignore-resources: a comma separated list of kubernetes resources that do not count as activity (e.g. pods, deployments.apps, secrets etc.)
• sleepmode.loft.sh/ignore-verbs: a comma separated list of kubernetes verbs that do not count as activity (e.g. get, list, create, update, patch, delete)
• sleepmode.loft.sh/ignore-resource-verbs: a comma separated list of resources and verbs that do not count as activity (format: myresource.mygroup=create update delete,myresource2.mygroup=create update)
• sleepmode.loft.sh/ignore-resource-names: a comma separated list of resources and names that do not count as activity (format: myresource.mygroup=name1 name2)
• sleepmode.loft.sh/ignore-active-conntections: if this annotation is set to "true", then still active kubernetes connections will be ignored for determining activity in a space

If you want to set some of this annotations automatically on space creation, you can add these to the space template in the account settings.

You can check when a namespace will start sleeping or will be deleted by hovering over its status in the Loft UI. Open connections describes how many keep alive connections are still active to that namespace (e.g. through kubectl exec).

Display information when a namespace starts sleeping

How can I specify when a namespace should sleep or should be deleted?

The period when a namespace should start sleeping can be configured through namespace annotations:

• sleepmode.loft.sh/sleep-after: tells Loft to trigger sleep mode automatically after x seconds of inactivity
• sleepmode.loft.sh/delete-after: tells Loft to delete the namespace automatically after x seconds of inactivity
• sleepmode.loft.sh/force: if "true" tells Loft to force sleep this namespace
• sleepmode.loft.sh/force-duration: the amount of seconds after the space starts sleeping after which no automatic wake up should occur (a value of 0 means infinite). This can be useful if there is an application accessing the space that would directly wake up the space again after it started sleeping.
• sleepmode.loft.sh/delete-all-pods: if "true" will delete all running pods within a namespace as soon as it starts sleeping, besides scaling down deployments, replicasets and statefulsets

Loft checks every space with an automatic sleep mode annotation in a fixed interval of 1 minute.

How to deal with long running requests

It's possible that certain requests such as kubectl exec or kubectl port-forward keep an active connection open to the space that prevents it from sleeping. This is in most cases wanted since the space shouldn't start sleeping when someone is still clearly using it, however there are cases where the connection might be idle (someone left the laptop open, but is not using it anymore) in which you want to terminate such connections and put the space to sleep.

There are multiple ways to approach this problem of active connections preventing a space from sleeping:

1. You can tell loft to timeout idle streaming connections (such as kubectl exec, kubectl port-forward etc.) after a certain time period with the annotation loft.sh/streaming-connection-idle-timeout: '3600' on a cluster. With this annotation set, loft will close connections automatically that are idle after the given seconds. By default, Loft will not timeout any connections. This will only apply to new opened connections to that cluster and not affect already running connections.

   

   Set annotation on cluster

2. Tell loft to ignore all active connections for determining space activity via the space annotation sleepmode.loft.sh/ignore-active-conntections: 'true'. This will put a space to sleep even though there still might be open connections such as kubectl exec or kubectl port-forward.

3. Configure your kubelets with the flag --streaming-connection-idle-timeout duration(see docs). This behaves essentially as option 1, however this configuration is Loft independent.

How can I prevent certain resources within a space from sleeping?

Loft allows you to specify resources that should not sleep within a space by providing the annotation sleepmode.loft.sh/exclude: 'true' on either a Deployment, StatefulSet, ReplicaSet or Pod. For example the following Deployment would not sleep if the space is sleeping:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  annotations:
    sleepmode.loft.sh/exclude: 'true'
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: busybox
          image: busybox
          command:
            - sleep
            - '3600'