Kubernetes Power Manager
This is an experimental project based on https://github.com/intel/kubernetes-power-manager (which has been
discontinued). It also includes enhancements from https://github.com/AMDEPYC/kubernetes-power-manager
(e.g. Dynamic CPU Frequency Management based on DPDK telemetry).
Introduction
The Kubernetes Power Manager is a Kubernetes Operator that has been developed to provide cluster users with a
Kubernetes native mechanism to configure power management settings (e.g. c-states, p-states, uncore) through CRDs.
The main features include the ability to:
- Configure per-CPU power management (p-states, c-states) for reserved CPUs, shared CPUs and application workload
CPUs (guaranteed pods) independently. - Modify per-CPU power management configuration at runtime (for reserved, shared or application CPUs).
- Configure processor level power management (e.g. uncore frequency for Intel processors).
- Modify p-states for guaranteed pod CPUs running DPDK applications based on DPDK metrics.
The Kubernetes Power Manager supports Intel, AMD and ARM processor architectures. Modern processors give users more
precise control over CPU performance and power use on a per-core basis. Yet, Kubernetes is purposefully built to
operate as an abstraction layer between the workload and such hardware capabilities as a workload orchestrator. Users
of Kubernetes who are running performance-critical workloads with particular requirements reliant on hardware
capabilities encounter a challenge as a consequence.
The Kubernetes Power Manager bridges the gap between the container orchestration layer and hardware features enablement.
Kubernetes Power Manager main responsibilities
- The Kubernetes Power Manager consists of two main components:
- the overarching manager which is deployed anywhere on a cluster
- the power node agent which is deployed on each node you require power management capabilities.
- The overarching operator is responsible for the configuration and deployment of the power node agent, while the power
node agent is responsible for the tuning of the cores as requested by the user.
Use Cases
- Power Optimization over Performance.
A user may be interested in fast response time, but not in maximal response time, so may choose to spin up cores on
demand but want to remain in power-saving mode the rest of the time. This can be done by configuring min/max CPU
frequency ranges and a CPUFreq governor that adjusts the frequency based on CPU usage. In addition, selected c-states
can be enabled to provide additional power savings when a CPU is idle. - Performance over Power Optimization.
A user may only be interested in fast response time, so may choose to have cores running at a high frequency at all
times and disable most or all c-states to avoid any latency penalties from waking a CPU. - Mixed use. A user may have a combination of applications - some of which demand the highest performance and
response time, and some that are more concerned with power optimization.
The Kubernetes Power Manager supports all of the above use cases.
Further Info: Please see the diagrams-docs directory for diagrams with a visual breakdown of the power manager and its components.
Functionality of the Kubernetes Power Manager
-
Frequency Tuning
Frequency tuning allows the individual cores on the system to be sped up or slowed down by changing their frequency.
This tuning is done via the Power Optimization Library which is now part of the project. More details in the kernel CPU Performance Scaling section.-
scaling_min_freqandscaling_max_freq- The
minandmaxvalues for a core are defined in thePowerProfileand the tuning is done after the core has been assigned by the Native CPU Manager. - The min/max frequency can be specified as an absolute value (in kHz) or as a linerar interpolation of hardware max and hardware min:
value = <hardware_min> + (hardware_max - hardware_min) * X%). - If min/max are not specified, hardware defaults will be used.The frequency of the cores are changed by writing the new frequency value to the
/sys/devices/system/cpu/cpuN/cpufreq/scaling_max|min_freqfile for the given core.
- The
-
Scaling Drivers
The following scaling drivers are currently supported in KPM:
-
intel_pstate
Modern Intel CPUs automatically employ the
intel_pstateCPU power scaling driver. This driver is integrated rather
than a module, giving it precedence over other drivers. For Sandy Bridge and newer CPUs, this driver is currently
used automatically. The BIOS P-State settings might be disregarded by Intel P-State.
The Intel P-State driver utilizes the Performance and Powersave governors.- Performance: The CPUfreq governor
performancesets the CPU statically to the highest frequency within the borders ofscaling_min_freqandscaling_max_freq. - Powersave: The CPUfreq governor
powersavesets the CPU statically to the lowest frequency within the borders ofscaling_min_freqandscaling_max_freq.
- Performance: The CPUfreq governor
-
acpi-cpufreq
The acpi-cpufreq driver setting operates much like the P-state driver but has a different set of available
governors. For more information see here.One thing to note is that acpi-cpufreq reports the base clock as the frequency hardware limits however the P-state
driver uses turbo frequency limits.
Both drivers can make use of turbo frequency; however, acpi-cpufreq can exceed hardware frequency limits when using
turbo frequency.
This is important to take into account when setting frequencies for profiles. -
intel_cpufreq
-
amd-pstate
-
amd-pstate-epp
-
cppc_cpufreq
This is often the default for aarch64 systems.
-
-
Energy Performance Preference (EPP)
The user can arrange cores according to priority levels using this capability. When the system has extra power, it can
be distributed among the cores according to their priority level. Although it cannot be guaranteed, the system will
try to apply the additional power to the cores with the highest priority. This feature requires support from both
the underlying processor and the scaling driver.
There are four levels of priority available:- Performance
- Balance Performance
- Balance Power
- Power
The Priority level for a core is defined using its EPP (Energy Performance Preference) value, which is one of the
options in the Power Profiles. If not all the power is utilized on the CPU, the CPU can put the higher priority cores
up to Turbo Frequency (allows the cores to run faster).
-
-
CPU Idle Time Management
To save energy on a system, you can allow the CPU to go into a low-power mode. Each CPU has several power modes, which are collectively called C-States. These work by cutting the clock signal and power from idle CPUs, or CPUs that are not executing commands. More details in the kernel CPU Idle section.
KPM supports both explicit C-state configuration by name and latency-based C-states configuration, allowing for a more fine-grained control over the trade-off between power saving and latency. C-states can now be configured directly within PowerProfiles alongside P-state settings.
- The C-States configuration in Linux is stored in
/sys/devices/system/cpu/cpuN/cpuidleor/sys/devices/system/cpu/cpuidle. To determine the driver in use, simply check the/sys/devices/system/cpu/cpuidle/current_driverfile. - Before configuring C-states in a PowerProfile, the user must confirm which C-states are actually available on the system. The available C-States are found under
/sys/devices/system/cpu/cpuN/cpuidle/stateN/.
- The C-States configuration in Linux is stored in
-
Uncore and equivalents
-
Intel
The largest part of modern CPUs is outside the actual cores. On Intel CPUs this is part is called the "Uncore" and has
last level caches, PCI-Express, memory controller, QPI, power management and other functionalities.
The previous deployment pattern was that an uncore setting was applied to sets of servers that are allocated as
capacity for handling a particular type of workload. This is typically a one-time configuration today. The Kubenetes
Power Manager now makes this dynamic and through a cloud native pattern. The implication is that the cluster-level
capacity for the workload can then configured dynamically, as well as scaled dynamically. Uncore frequency applies to
Xeon scalable and D processors could save up to 40% of CPU power or improved performance gains. -
AMD
Unlike Intel, AMD does not expose uncore frequency controls (such as LLC, memory controller, or fabric clocks) via a
standard kernel interface. There is no equivalent to Intel’sintel_uncore_frequencydriver or
/sys/devices/system/cpu/intel_uncore_frequencysysfs interface.Instead, uncore frequency management on AMD EPYC platforms is supported via the
ESMI library, a user-space interface that communicates with
hardware using the amd_hsmp kernel driver.Note: The amd_hsmp driver might not be loaded by default and must be manually enabled:
sudo modprobe amd_hsmpIn ADM KPM, the following logic applies when configuring DF P-states:
- When min equals max, a fixed DF P-state is set. This disables automatic DF p-state scaling and locks the DF to operate at that specific performance level.
- When min differs max, DF is allowed to dynamically scale between the specified DF P-states range.
This is not currently supported in KPM.
-
ARM - no equivalent supported
-
Prerequisites
-
Node Feature Discovery (NFD) should be deployed in
the cluster before running the Kubernetes Power Manager.
NFD is used to detect node-level features such as Intel Speed Select Technology - Base Frequency (SST-BF).
Once detected, the user can instruct the Kubernetes Power Manager to deploy the Power Node Agent to Nodes with
SST-specific labels, allowing the Power Node Agent to take advantage of such features by configuring cores on the
host to optimise performance for containerized workloads.Note: NFD is recommended, but not essential. Node labels can also be applied manually. See
the NFD repo for a full list of features
labels. -
If not using NFD or labels added through NFD, label the node manually with a label of your choosing:
kubectl label node <node-name> feature.node.kubernetes.io/power-node=trueNote: Make sure to use the same label in the
PowerConfig, underspec.powerNodeSelector. -
Important: In the kubelet configuration file the
cpuManagerPolicyhas to set tostatic, and the
reservedSystemCPUsmust be set to the desired value (full file here):apiVersion: kubelet.config.k8s.io/v1beta1 ... cpuManagerPolicy: "static" ... reservedSystemCPUs: "0" ...
Deploying the Kubernetes Power Manager using kustomize
-
Build the 2 images:
IMG_AGENT=quay.io/<user/org>/power-node-agent:latest IMG=quay.io/<user/org>/power-operator:latest IMGTOOL=<docker/podman> make update OCP=true IMG_AGENT=quay.io/<user/org>/power-node-agent:latest IMG=quay.io/<user/org>/power-operator:latest IMGTOOL=<docker/podman> make images-ocp
Note: By default, the images are built for x86_64 platforms. For an ARM platform, add the
PLATFORM=linux/arm64parameter:OCP=true IMG_AGENT=<...> IMG=<...> PLATFORM=linux/arm64 IMGTOOL=<...> make images-ocp -
Push the 2 images:
<docker/podman> push <image> -
Install the CRDs and deploy the operator:
OCP=true IMG_AGENT=quay.io/<user/org>/power-node-agent:latest IMG=quay.io/<user/org>/power-operator:latest make install deploy
Building Multi-Architecture Images
The Kubernetes Power Manager supports building multi-architecture container images for both the Power Operator and Power Node Agent. This allows you to create a single image tag that works across multiple architectures (e.g., AMD64 and ARM64).
Prerequisites
For Podman:
Podman 3.0+ with native multi-arch support:
# Verify podman version (3.0+ required)
podman version
# On Linux: Ensure qemu-user-static is installed for cross-platform builds
sudo apt-get install qemu-user-static # Debian/Ubuntu
sudo dnf install qemu-user-static # Fedora/RHEL
# On macOS: Initialize and start podman machine
# Make sure Rosetta is enabled when building on Apple Silicon
podman machine init # customize cpus, disk-size, memory if required
podman machine startFor Docker:
Docker buildx must be installed and configured:
# Check if buildx is available
docker buildx version
# Create a new builder instance (one-time setup)
docker buildx create --name multiarch --use
docker buildx inspect --bootstrapBuild Multi-Arch Images
Build both operator and agent images for multiple architectures (default: linux/amd64 and linux/arm64):
# Using Podman
IMGTOOL=podman \
IMAGE_REGISTRY=quay.io/<user/org> \
VERSION=latest \
make build-push-multiarch
# Using Docker
IMGTOOL=docker \
IMAGE_REGISTRY=quay.io/<user/org> \
VERSION=latest \
make build-push-multiarchFor OpenShift (uses UBI base image and OCP-specific manifests):
OCP=true \
IMGTOOL=podman \
IMAGE_REGISTRY=quay.io/<user/org> \
VERSION=latest \
make build-push-multiarchCustomizing Target Platforms
Override the default platforms (linux/amd64,linux/arm64):
PLATFORMS=linux/amd64,linux/arm64,linux/arm/v7 \
IMGTOOL=podman \
IMAGE_REGISTRY=quay.io/<user/org> \
VERSION=latest \
make build-push-multiarchVerifying Multi-Arch Images
After pushing, verify the multi-arch manifest:
# Using Podman
podman manifest inspect quay.io/<user/org>/kubernetes-power-manager-operator:latest
# Using Docker
docker buildx imagetools inspect quay.io/<user/org>/kubernetes-power-manager-operator:latestDeploying the Kubernetes Power Manager using Helm
The Kubernetes Power Manager includes a helm chart for the latest releases, allowing the user to easily deploy
everything that is needed for the overarching operator and the node agent to run. The following versions are
supported with helm charts:
- TODO
When set up using the provided helm charts, the following will be deployed:
- The power-manager namespace
- The RBAC rules for the operator and node agent
- The operator deployment itself
- The operator's power config
- A shared power profile
To change any of the values the above are deployed with, edit the values.yaml file of the relevant helm chart.
To deploy the Kubernetes Power Manager using Helm, you must have Helm installed. For more information on installing
Helm, see the installation guide here.
To install the latest version, use the following command:
make helm-install
To uninstall the latest version, use the following command:
make helm-uninstall
You can use the HELM_CHART and OCP parameters to deploy an older or Openshift specific version of the Kubernetes Power Manager:
HELM_CHART=v2.3.1 OCP=true make helm-install
HELM_CHART=v2.2.0 make helm-install
HELM_CHART=v2.1.0 make helm-install
Please note when installing older versions that certain features listed in this README may not be supported.
Components
Power Optimization Library
The Power Optimization Library takes the desired configuration
for the cores associated with Exclusive Pods and tunes them based on the requested PowerProfile. The Power Optimization
Library will also facilitate the use of C-States functionality.
Power Node Agent
The Power Node Agent is also a containerized application deployed by the Kubernetes Power Manager in a DaemonSet.
The primary function of the node agent is to communicate with the node's Kubelet PodResources endpoint to discover the
exact cores that are allocated per container. The node agent watches for Pods that are created in your cluster and examines
them to determine which PowerProfile they have requested and then sets off the chain of events that tunes the
frequencies of the cores designated to the Pod.
Power Config controller
The Kubernetes Power Manager will wait for the PowerConfig CR to be created by the user to initiate the deployment of
the node agent. The PowerConfig specifies what Nodes the user wants to place the node agent on.
spec.powerNodeSelector: This is a key/value map used for defining a list of node labels that a node must satisfy in order for the Power Node Agent to be deployed.
Once the Power Config controller sees that the PowerConfig is created, it deploys the power node agent on each of the
Nodes that are specified. PowerProfiles should be created separately by the user and are advertised as
extended resources that can be requested in the PodSpec. The Kubelet can then keep track of these requests.
The extended resources can control how many cores on the system can be run at a higher frequency and help avoid hitting
the heat threshold which would limit frequencies.
Note: Only one PowerConfig can be present in a cluster. The Config Controller will ignore and delete and subsequent
PowerConfigs created after the first.
Example:
apiVersion: "power.openshift.io/v1"
kind: PowerConfig
metadata:
name: power-config
namespace: power-manager
spec:
powerNodeSelector:
feature.node.kubernetes.io/power-node: "true"Power Workload controller
The Power Workload controller is responsible for the actual tuning of the cores. The Power Workload Controller uses the
Power Optimization Library and requests that it creates the Pools. The Pools hold the PowerProfile associated with the
cores and the cores that need to be configured.
PowerWorkload objects are automatically created for each valid non-Shared PowerProfile by the Power Profile controller.
PowerWorkload objects can also be created directly by the user via the PowerWorkload spec. This is only recommended
when creating the Shared PowerWorkload for a given Node, as this is the responsibility of the user. If no Shared
PowerWorkload is created, the cores that remain in the shared pool on the Node will remain at their core frequency
values instead of being tuned to lower frequencies. PowerWorkloads are specific to a given node, so one is created for
each Node with a Pod requesting a PowerProfile, based on the PowerProfile requested.
Note: A Shared
PowerWorkloadis mandatory if guaranteed pods will need to usePowerProfiles.
Example PowerWorkload with exclusive CPUs:
apiVersion: "power.openshift.io/v1"
kind: PowerWorkload
metadata:
name: performance-example-node-workload
namespace: power-manager
spec:
name: "performance-example-node-workload"
powerProfile: "performance-example-node"
status:
workloadNodes:
containers:
- exclusiveCPUs:
- 2
- 3
- 66
- 67
id: f1be89f7dda457a7bb8929d4da8d3b3092c9e2a35d91065f1b1c9e71d19bcd4f
name: example-container
pod: example-pod
powerProfile: “performance-example-node”
name: “example-node”
cpuIds:
- 2
- 3
- 66
- 67This workload assigns the performance PowerProfile to cores 2, 3, 66, and 67 on the node example-node.
The Shared PowerWorkload created by the user is determined by the power workload controller based on the allCores
value in the PowerWorkload spec.
The reserved CPUs on the Node must also be specified, as these will not be considered for frequency tuning by the
controller as they are always being used by Kubernetes’ processes.
It is important that the reservedCPUs value directly corresponds to the reservedCPUs value in the user’s Kubelet config to
keep them consistent.
The user determines the Node for this PowerWorkload using the spec.powerNodeSelector to match the labels on the Node.
The user then specifies the requested PowerProfile to use.
The shared PowerWorkload must select a unique node through its spec.powerNodeSelector (create one shared
PowerWorkload per node in multi-node clusters), so it is recommended that the kubernetes.io/hostname label be used.
A shared PowerProfile can be used for multiple shared PowerWorkloads.
Example:
apiVersion: "power.openshift.io/v1"
kind: PowerWorkload
metadata:
name: shared-example-node-workload
namespace: power-manager
spec:
name: "shared-example-node-workload"
allCores: true
reservedCPUs:
- cores: [0, 1]
powerProfile: "performance"
powerNodeSelector:
# Labels other than hostname can be used
- “kubernetes.io/hostname”: “example-node”
powerProfile: "shared-example-node"Power Profile Controller
The Power Profile controller holds values for specific settings which are then applied to cores at host level by the
Kubernetes Power Manager as requested. PowerProfiles are advertised as extended resources and can be requested via the
PodSpec. All PowerProfiles must be created explicitly by the user.
Example:
apiVersion: "power.openshift.io/v1"
kind: PowerProfile
metadata:
name: performance-example-application
spec:
name: "performance-example-application"
cpuCapacity: "75%"
nodeSelector:
labelSelector:
matchExpressions:
- key: test
operator: In
values:
- test
pstates:
max: 3700
min: 3300
epp: "performance"
cstates:
# maxLatencyUs: 100
names:
C1: true
C6: falseNote:
spec.pstates.minandspec.pstates.maxcan hold both scalar and percentage values.spec.cpuCapacityhas been added to configure the node's CPU capacity. It can hold both scalar and percentage values.spec.nodeSelectorcan be used to choose to which node thePowerProfileapplies to.- The PowerProfile CRD has been enhanced to support both P-states (frequency) and C-states (power saving) configuration in
a single, unified structure. C-states can be configured either by explicit state names or by maximum latency threshold
for more flexible power tuning across different CPU architectures. - The
spec.pstates.epponly applies to processors that support it.
Dynamic scaling for DPDK polling workloads is also supported via spec.cpuScalingPolicy.
See Dynamic CPU Frequency Scaling for DPDK workloads for details.
The Shared PowerProfile must also be created by the user and does not require a Base PowerProfile. This allows the user
to have a Shared PowerProfile per Node in their cluster, giving more room for different configurations. The Power
controller determines that a PowerProfile is being designated as Shared through the use of the spec.shared parameter.
This flag must be enabled when using a shared pool.
Examples:
apiVersion: "power.openshift.io/v1"
kind: PowerProfile
metadata:
name: shared-example-node1
spec:
name: "shared-example-node1"
shared: true
pstates:
max: 1500
min: 1000
epp: "power"
governor: "powersave"apiVersion: "power.openshift.io/v1"
kind: PowerProfile
metadata:
name: shared-example-node2
spec:
name: "shared-example-node2"
cpuCapacity: "75%"
nodeSelector:
labelSelector:
matchExpressions:
- key: test
operator: In
values:
- test
shared: true
pstates:
max: "20%" # scalar is also accepted; if missing, it defaults to the hardware max
min: "50%" # scalar is also accepted; if missing, it defaults to the hardware max
governor: "powersave"
# Names or maxLatencyUs, only one is supported.
cstates:
# maxLatencyUs: 100
names:
C1: true
C6: falseNote: The Power Profile controller will create a corresponding
PowerWorkloadfor each valid
non-SharedPowerProfile.
Power Node controller
The Power Node controller provides observability into the node's power management operations. It displays the current state of PowerProfiles, PowerWorkloads, core assignments, and guaranteed pod containers.
Example Status:
status:
powerContainers:
- exclusiveCpus:
- 2
- 26
id: cri-o://e91dca501a08c343bb18e42e86d3d6c9e146db50782fe56ec5fe776160dc43dc
name: example-container
pod: example-pod
powerProfile: balance-performance
workload: performance-example-node-workload
powerProfiles:
- 'balance-performance: 2900000 || 2700000 || balance_performance || powersave ||
enabled: C1,C1E,POLL; disabled: C6'
- 'balance-power: 2200000 || 2000000 || balance_power || powersave || enabled: C1,C1E,POLL;
disabled: C6'
- 'performance: 95% || 75% || performance || powersave || enabled: C1,C1E,POLL;
disabled: C6'
- 'shared-power-saving: 1800000 || 800000 || power || powersave || enabled: C1,C1E,C6,POLL'
powerWorkloads:
- 'balance-power: balance-power || '
- 'performance: performance || '
- 'balance-performance: balance-performance || 2,26'
reservedPools:
- 3600000 || 3400000 || 0-1,24-25
sharedPool: shared-power-saving || 1800000 || 800000 || 3-23,27-47The status shows:
- powerProfiles: Available profiles with frequencies, EPP, governor, and C-state configuration
- powerWorkloads: Active workloads with their assigned cores
- powerContainers: Guaranteed pod containers with exclusive core assignments
- sharedPool: Shared pool profile and core assignments
- reservedPools: Reserved platform cores with custom profiles
Power Pod controller
The Power Pod Controller watches for pods. When a pod comes along the Power Pod Controller checks if the pod is in the
guaranteed quality of service class (using exclusive
cores, see documentation, taking a core
out of the shared pool as it is the only option in Kubernetes that can do this operation). Then it examines the Pods to
determine which PowerProfile has been requested and then updates the appropriate PowerWorkload.
Note: the request and the limits must have a matching number of cores and are also on a container-by-container basis.
Currently the Kubernetes Power Manager supports multiplePowerProfileper Pod, but only onePowerProfileper container.
Uncore Frequency - only applicable to Intel CPUs
Error handling
If any error occurs it will be displayed in the status field of the custom resource, for example:
apiVersion: power.openshift.io/v1
kind: PowerProfile
...
status:
errors:
- the PowerProfile CRD name must match name of one of the power nodesIf no errors occurred or were corrected, the list will be empty
apiVersion: power.openshift.io/v1
kind: PowerProfile
...
status:
errors: []End to end workflow
-
Label the nodes in accordance to the
spec.powerNodeSelectorfield of thePowerConfigthat will be applied. -
Create the Power Config CR - use the example PowerConfig.
kubectl apply -f examples/example-powerconfig.yamlOnce deployed, the controller-manager pod will see it via the PowerConfig controller and create a Node Agent instance on nodes specified with the
feature.node.kubernetes.io/power-node: "true"label.The power-node-agent DaemonSet will be created, managing the Power Node Agent Pods.
-
Create the Shared PowerProfile - use the example Shared PowerProfile.
kubectl apply -f examples/example-shared-profile.yaml -
If needed, create other non-Shared
PowerProfiles.Note: These would usually be used for reserved or exclusive CPUs.
-
Create the Shared PowerWorkload - use the example Shared PowerWorkload.
Replace the necessary values with those that correspond to your cluster and apply the Workload:
kubectl apply -f examples/example-shared-workload.yaml- Once applied, the PowerWorkload controller will get triggered and create the corresponding Pool.
- All of the cores on the system except for the reservedCPUs will then be brought down to this lower frequency level.
- The reservedCPUs will be kept at the system default min and max frequency by default. If the user specifies a
PowerProfilealong with a set of reserved
cores then a separate pool will be created for those cores and that profile. If an invalid profile is supplied the
cores will instead be placed in the default reserved pool with system defaults.Note: In most instances, leaving these cores at system defaults is the best approach to prevent important k8s or kernel related processes from becoming starved.
-
Create the Performance Pod(s) - use the example Pod.
Replace the placeholder values with the
PowerProfileyou require and apply:kubectl apply -f examples/example-pod.yamlAt this point, if only the performance
PowerProfilehas been created, the cluster will contain 2 PowerProfiles and 2 PowerWorkloads:# kubectl get powerprofiles -n power-manager NAME AGE performance 59m shared-<NODE_NAME> 60m # kubectl get powerworkloads -n power-manager NAME AGE performance-<NODE_NAME>-workload 63m shared-<NODE_NAME>-workload 61m
Check the
PowerNodestatus CR for a summary of the KPM resources and the CPU configuration. -
Delete Pods
kubectl delete pods <name>When a Pod that was associated with a
PowerWorkloadis deleted, the cores associated with that Pod will be removed from the correspondingPowerWorkload.
All cores removed from the PowerWorkload are added back to the Shared PowerWorkload for that Node and returned to the lower frequencies.