Professional Documents
Culture Documents
1 of 3
https://www.ibm.com/developerworks/community/blogs/aixpert/entry/...
IBM
Sign in | Register
Technical topics
Evaluation software
Community
Blogs
Events
Search developerWorks
This Blog
Search
AIXpert Blog
Share
Tweet
Related posts
For Capped virtual machines, the Entitlement is the maximum of compute cycles that can be used.
For Uncapped virtual machines and if there are "spare" CPU cycles in the Shared CPU Pool (or other
virtual machines are not requiring their full entitlement) then the VP number is the limit of the total number
Updated July 10
Updated June 18
PureApplication Syst...
Updated June 13
Links
YouTube vidoes on AIX, POWER7,...
Tags
3. No longer scheduling the virtual machine to a CPU it does not need just to perform pointless context
Find a Tag
switches to the virtual machine to then give up the CPU milliseconds later = more work done.
Detecting folding: We also noted that these folded virtual processors do actually get some CPU cycles. nmon
used the libperfstat AIX library to extract the performance numbers and we found that instead of billions of clock
ticks on the processor these folded ones have a few - in the 50 to 100 ticks sort of range. We think this is a tiny
amount of house keeping going on like collecting the processor hardware stats, perhaps a clock interrupt and I
think that if a processor starts a device driver to action an adapter to perform an operation that the later interrupt
is returned to the same processor (which might now be folded). This is a little guess work on my part so don't
aix
director
hmc
aix7
hypervisor linux
lpar
memory
quote me - please! Back to nmon: there is no official AIX folding statistics from any interface that I have come
across. nmon deduces the folding number by monitoring the physical clock ticks and heuristically (I like that
power
words) determines if it is folded or not. In C programming terms, are the number of ticks below a threshold we
systems
worked out empirically (another good word) by sitting there and watching the numbers while tweaking the
aix6
power6
power7
powervm
vm workload
vios virtual
wpar
List
Now for the complicated bit - SMT! When a virtual processor is folded all the Simultaneous MultiThreading
(SMT) threads are switched off together - the different threads are always running in the same virtual machine.
This made the heuristics in the nmon code tricky as it has to check all the threads on the same CPU are doing
Recent tweets
practically nothing. This was written when Power6 was the current processor and SMT=2 was the maximum. Of
course, when Power7 came along with SMT=4 and the code had to be reworked - in fact there is a nmon release
that runs on Power7 where the code had not been updated and gets the folding count hopelessly wrong but its
Follow @mr_nmon
fairly obvious when you see it (you can't fold away more CPUs than you actually have!) - install a service pack to
fix this.
What does AIX actually do when it folds? We have seen that the virtual processors are still there and seen by
AIX and occasionally running a few clock cycles. So they are not stopped or released. AIX is simply no longer
scheduling processes to run on them. This gives the hypervisor a hint that at the moment it does not have to
schedule the virtual machine on them. The AIX kernel to Hypervisor interface and mechanisms are definitely a
secret, so I have no clues at this level.
7/13/2015 8:42 PM
2 of 3
https://www.ibm.com/developerworks/community/blogs/aixpert/entry/...
When to fold? AIX does this slowly but seems to monitor the CPU use for a period of time (from my
observations a few seconds which is a long long time in CPU terms) and determines that the CPU use is steady
or dropping and its is safe and efficient to fold a CPU off. It then waits to see what happens next and may then
fold away a further CPU. It is maths that larger virtual machines with many 100's or 1000's of active processes
and threads of execution do not have sudden jumps of CPU requirements but the workloads sort of "flows in and
out" and CPU use is smoothed out.
When to unfold? AIX again monitors (it is probably the same algorithm) the CPU use. When it notices a
consistent high use of the current unfolded CPUs it decides that unfolding could help the throughput of the
processes and unfolds a CPU. If there is a sudden peak in runnable processes it does not immediately unfold.
This is because it can already deal successfully with short term transitory peaks as normal via the run queues. If
fact, over aggressive unfolding could slow the applications down. When a CPU is unfolded its caches will be
empty then a happy process on the running CPUs with hot cache is moved to the new unfolded one with cold
caches - it will spend the next few milliseconds loading the cache with program, data, stack and heap memory
lines before it can get back to full speed. If we then find the temporary peak has passed, AIX will fold the CPU
again - it was all rather a waste of time and the process moved twice actually got slower. This is why it holds off a
little time before unfolding and makes sure it is a genuine growth in the demand for CPU time. It gives me the
impression that, for slowly growing workloads, AIX unfolds at something like a virtual processors once a second
but that is an observation rather than a fact.
Why are we monitoring Folding? Well, I was keen to have this in nmon because it gives us good clues about
whether we have the right Entitlement for our virtual machine. If (particularly monitoring long term like over a
month), we always find that we have Folded virtual processors and particularly during our peaks in workload then
perhaps we have the Entitlement set too high. It could be dropped to let other workloads be added to the same
machine. On the other hand, if we have a critical production virtual machines that has for long peak periods
Folding = zero then perhaps we should consider raising the Entitlement to make sure this virtual machine always
has sufficient CPU cycles.
Folding is Leaking out! This advanced optimisation technique inside AIX was an internal secret at one time but
has become known. I suspect my nmon monitoring might have accelerates that a little :-) But there are a few
comments in the manual now and there are a few Folding tuning parameters available.
The advanced AIX scheduling tuning command, "schedo" has these options:
AIX6+ vpm_fold_policy
AIX6+ vpm_xvcpus
AIX6 TL06+ & AIX 7 also has vpm_fold_threshold (in the schedo command but not in the
documentation yet). This is the utilisation threshold at which AIX begins to unfold. In the past this was
80% and in the latest AIX release been lowered.
I would not go fiddling with these unless you have a particular problem to fix and have recommendations
fro AIX support. You can easily make performance worse with non-default settings. Check the current
settings and that try are the default with (as the root user): schedo -L
In the manual pages:
Virtual Processor discussion in the manuals
mpstat -s 1 9999 is an alternative way to "see" Folded virtual CPUs.
Read the AIX online manual for more information on these.
I hope this helps, thanks, Nigel Griffiths.
Tags: power7 powervm power6 hypervisor aix folding aix6 aix7 virtual processor
Add a Comment
More Actions
Comments (4)
RWolter commented Apr 13
Add a Comment
More Actions
Comment Permalink
Comment Permalink
Comment Permalink
7/13/2015 8:42 PM
3 of 3
https://www.ibm.com/developerworks/community/blogs/aixpert/entry/...
Comment Permalink
Excellent post. One of the better explanations of processor folding. Thanks Nigel.
Previous Entry
Main
Next Entry
About
Feeds
Report abuse
Faculty
Select a language:
Help
Newsletters
Terms of use
Students
English
Contact us
Business Partners
Submit content
IBM privacy
IBM accessibility
Portugus (Brasil)
Espaol
Vit
7/13/2015 8:42 PM