(split) LDP: Release pages for LDP v3.39.

[linuxjm/LDP_man-pages.git] / release / man7 / cpuset.7

.\" Copyright (c) 2008 Silicon Graphics, Inc.
.\"
.\" Author: Paul Jackson (http://oss.sgi.com/projects/cpusets)
.\"
.\" This is free documentation; you can redistribute it and/or
.\" modify it under the terms of the GNU General Public License
.\" version 2 as published by the Free Software Foundation.
.\"
.\" The GNU General Public License's references to "object code"
.\" and "executables" are to be interpreted as the output of any
.\" document formatting or typesetting system, including
.\" intermediate and printed output.
.\"
.\" This manual is distributed in the hope that it will be useful,
.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
.\" GNU General Public License for more details.
.\"
.\" You should have received a copy of the GNU General Public
.\" License along with this manual; if not, write to the Free
.\" Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
.\" MA 02111, USA.
.\"
.\"*******************************************************************
.\"
.\" This file was generated with po4a. Translate the source file.
.\"
.\"*******************************************************************
.TH CPUSET 7 2008\-11\-12 Linux "Linux Programmer's Manual"
.SH 名前
cpuset \- confine processes to processor and memory node subsets
.SH 説明
The cpuset file system is a pseudo\-file\-system interface to the kernel
cpuset mechanism, which is used to control the processor placement and
memory placement of processes.  It is commonly mounted at \fI/dev/cpuset\fP.
.PP
On systems with kernels compiled with built in support for cpusets, all
processes are attached to a cpuset, and cpusets are always present.  If a
system supports cpusets, then it will have the entry \fBnodev cpuset\fP in the
file \fI/proc/filesystems\fP.  By mounting the cpuset file system (see the
\fBEXAMPLE\fP section below), the administrator can configure the cpusets on a
system to control the processor and memory placement of processes on that
system.  By default, if the cpuset configuration on a system is not modified
or if the cpuset file system is not even mounted, then the cpuset mechanism,
though present, has no affect on the system's behavior.
.PP
A cpuset defines a list of CPUs and memory nodes.
.PP
The CPUs of a system include all the logical processing units on which a
process can execute, including, if present, multiple processor cores within
a package and Hyper\-Threads within a processor core.  Memory nodes include
all distinct banks of main memory; small and SMP systems typically have just
one memory node that contains all the system's main memory, while NUMA
(non\-uniform memory access) systems have multiple memory nodes.
.PP
Cpusets are represented as directories in a hierarchical pseudo\-file system,
where the top directory in the hierarchy (\fI/dev/cpuset\fP)  represents the
entire system (all online CPUs and memory nodes)  and any cpuset that is the
child (descendant) of another parent cpuset contains a subset of that
parent's CPUs and memory nodes.  The directories and files representing
cpusets have normal file\-system permissions.
.PP
Every process in the system belongs to exactly one cpuset.  A process is
confined to only run on the CPUs in the cpuset it belongs to, and to
allocate memory only on the memory nodes in that cpuset.  When a process
\fBfork\fP(2)s, the child process is placed in the same cpuset as its parent.
With sufficient privilege, a process may be moved from one cpuset to another
and the allowed CPUs and memory nodes of an existing cpuset may be changed.
.PP
When the system begins booting, a single cpuset is defined that includes all
CPUs and memory nodes on the system, and all processes are in that cpuset.
During the boot process, or later during normal system operation, other
cpusets may be created, as subdirectories of this top cpuset, under the
control of the system administrator, and processes may be placed in these
other cpusets.
.PP
Cpusets are integrated with the \fBsched_setaffinity\fP(2)  scheduling affinity
mechanism and the \fBmbind\fP(2)  and \fBset_mempolicy\fP(2)  memory\-placement
mechanisms in the kernel.  Neither of these mechanisms let a process make
use of a CPU or memory node that is not allowed by that process's cpuset.
If changes to a process's cpuset placement conflict with these other
mechanisms, then cpuset placement is enforced even if it means overriding
these other mechanisms.  The kernel accomplishes this overriding by silently
restricting the CPUs and memory nodes requested by these other mechanisms to
those allowed by the invoking process's cpuset.  This can result in these
other calls returning an error, if for example, such a call ends up
requesting an empty set of CPUs or memory nodes, after that request is
restricted to the invoking process's cpuset.
.PP
Typically, a cpuset is used to manage the CPU and memory\-node confinement
for a set of cooperating processes such as a batch scheduler job, and these
other mechanisms are used to manage the placement of individual processes or
memory regions within that set or job.
.SH FILES
Each directory below \fI/dev/cpuset\fP represents a cpuset and contains a fixed
set of pseudo\-files describing the state of that cpuset.
.PP
New cpusets are created using the \fBmkdir\fP(2)  system call or the
\fBmkdir\fP(1)  command.  The properties of a cpuset, such as its flags,
allowed CPUs and memory nodes, and attached processes, are queried and
modified by reading or writing to the appropriate file in that cpuset's
directory, as listed below.
.PP
The pseudo\-files in each cpuset directory are automatically created when the
cpuset is created, as a result of the \fBmkdir\fP(2)  invocation.  It is not
possible to directly add or remove these pseudo\-files.
.PP
A cpuset directory that contains no child cpuset directories, and has no
attached processes, can be removed using \fBrmdir\fP(2)  or \fBrmdir\fP(1).  It is
not necessary, or possible, to remove the pseudo\-files inside the directory
before removing it.
.PP
The pseudo\-files in each cpuset directory are small text files that may be
read and written using traditional shell utilities such as \fBcat\fP(1), and
\fBecho\fP(1), or from a program by using file I/O library functions or system
calls, such as \fBopen\fP(2), \fBread\fP(2), \fBwrite\fP(2), and \fBclose\fP(2).
.PP
.\" ====================== tasks ======================
The pseudo\-files in a cpuset directory represent internal kernel state and
do not have any persistent image on disk.  Each of these per\-cpuset files is
listed and described below.
.TP 
\fItasks\fP
List of the process IDs (PIDs) of the processes in that cpuset.  The list is
formatted as a series of ASCII decimal numbers, each followed by a newline.
A process may be added to a cpuset (automatically removing it from the
cpuset that previously contained it) by writing its PID to that cpuset's
\fItasks\fP file (with or without a trailing newline.)

.\" =================== notify_on_release ===================
\fBWarning:\fP only one PID may be written to the \fItasks\fP file at a time.  If
a string is written that contains more than one PID, only the first one will
be used.
.TP 
\fInotify_on_release\fP
.\" ====================== cpus ======================
Flag (0 or 1).  If set (1), that cpuset will receive special handling after
it is released, that is, after all processes cease using it (i.e., terminate
or are moved to a different cpuset)  and all child cpuset directories have
been removed.  See the \fBNotify On Release\fP section, below.
.TP 
\fIcpus\fP
List of the physical numbers of the CPUs on which processes in that cpuset
are allowed to execute.  See \fBList Format\fP below for a description of the
format of \fIcpus\fP.

.\" ==================== cpu_exclusive ====================
The CPUs allowed to a cpuset may be changed by writing a new list to its
\fIcpus\fP file.
.TP 
\fIcpu_exclusive\fP
Flag (0 or 1).  If set (1), the cpuset has exclusive use of its CPUs (no
sibling or cousin cpuset may overlap CPUs).  By default this is off (0).
Newly created cpusets also initially default this to off (0).

.\" ====================== mems ======================
Two cpusets are \fIsibling\fP cpusets if they share the same parent cpuset in
the \fI/dev/cpuset\fP hierarchy.  Two cpusets are \fIcousin\fP cpusets if neither
is the ancestor of the other.  Regardless of the \fIcpu_exclusive\fP setting,
if one cpuset is the ancestor of another, and if both of these cpusets have
nonempty \fIcpus\fP, then their \fIcpus\fP must overlap, because the \fIcpus\fP of
any cpuset are always a subset of the \fIcpus\fP of its parent cpuset.
.TP 
\fImems\fP
.\" ==================== mem_exclusive ====================
List of memory nodes on which processes in this cpuset are allowed to
allocate memory.  See \fBList Format\fP below for a description of the format
of \fImems\fP.
.TP 
\fImem_exclusive\fP
Flag (0 or 1).  If set (1), the cpuset has exclusive use of its memory nodes
(no sibling or cousin may overlap).  Also if set (1), the cpuset is a
\fBHardwall\fP cpuset (see below.)  By default this is off (0).  Newly created
cpusets also initially default this to off (0).

.\" ==================== mem_hardwall ====================
Regardless of the \fImem_exclusive\fP setting, if one cpuset is the ancestor of
another, then their memory nodes must overlap, because the memory nodes of
any cpuset are always a subset of the memory nodes of that cpuset's parent
cpuset.
.TP 
\fImem_hardwall\fP (since Linux 2.6.26)
.\" ==================== memory_migrate ====================
Flag (0 or 1).  If set (1), the cpuset is a \fBHardwall\fP cpuset (see below.)
Unlike \fBmem_exclusive\fP, there is no constraint on whether cpusets marked
\fBmem_hardwall\fP may have overlapping memory nodes with sibling or cousin
cpusets.  By default this is off (0).  Newly created cpusets also initially
default this to off (0).
.TP 
\fImemory_migrate\fP (since Linux 2.6.16)
.\" ==================== memory_pressure ====================
Flag (0 or 1).  If set (1), then memory migration is enabled.  By default
this is off (0).  See the \fBMemory Migration\fP section, below.
.TP 
\fImemory_pressure\fP (since Linux 2.6.16)
.\" ================= memory_pressure_enabled =================
A measure of how much memory pressure the processes in this cpuset are
causing.  See the \fBMemory Pressure\fP section, below.  Unless
\fImemory_pressure_enabled\fP is enabled, always has value zero (0).  This file
is read\-only.  See the \fBWARNINGS\fP section, below.
.TP 
\fImemory_pressure_enabled\fP (since Linux 2.6.16)
.\" ================== memory_spread_page ==================
Flag (0 or 1).  This file is only present in the root cpuset, normally
\fI/dev/cpuset\fP.  If set (1), the \fImemory_pressure\fP calculations are enabled
for all cpusets in the system.  By default this is off (0).  See the
\fBMemory Pressure\fP section, below.
.TP 
\fImemory_spread_page\fP (since Linux 2.6.17)
.\" ================== memory_spread_slab ==================
Flag (0 or 1).  If set (1), pages in the kernel page cache (file\-system
buffers) are uniformly spread across the cpuset.  By default this is off (0)
in the top cpuset, and inherited from the parent cpuset in newly created
cpusets.  See the \fBMemory Spread\fP section, below.
.TP 
\fImemory_spread_slab\fP (since Linux 2.6.17)
.\" ================== sched_load_balance ==================
Flag (0 or 1).  If set (1), the kernel slab caches for file I/O (directory
and inode structures) are uniformly spread across the cpuset.  By default
this is off (0) in the top cpuset, and inherited from the parent cpuset in
newly created cpusets.  See the \fBMemory Spread\fP section, below.
.TP 
\fIsched_load_balance\fP (since Linux 2.6.24)
.\" ================== sched_relax_domain_level ==================
Flag (0 or 1).  If set (1, the default) the kernel will automatically load
balance processes in that cpuset over the allowed CPUs in that cpuset.  If
cleared (0) the kernel will avoid load balancing processes in this cpuset,
\fIunless\fP some other cpuset with overlapping CPUs has its
\fIsched_load_balance\fP flag set.  See \fBScheduler Load Balancing\fP, below, for
further details.
.TP 
\fIsched_relax_domain_level\fP (since Linux 2.6.26)
.\" ================== proc cpuset ==================
Integer, between \-1 and a small positive value.  The
\fIsched_relax_domain_level\fP controls the width of the range of CPUs over
which the kernel scheduler performs immediate rebalancing of runnable tasks
across CPUs.  If \fIsched_load_balance\fP is disabled, then the setting of
\fIsched_relax_domain_level\fP does not matter, as no such load balancing is
done.  If \fIsched_load_balance\fP is enabled, then the higher the value of the
\fIsched_relax_domain_level\fP, the wider the range of CPUs over which
immediate load balancing is attempted.  See \fBScheduler Relax Domain Level\fP,
below, for further details.
.PP
.\" ================== proc status ==================
In addition to the above pseudo\-files in each directory below
\fI/dev/cpuset\fP, each process has a pseudo\-file,
\fI/proc/<pid>/cpuset\fP, that displays the path of the process's
cpuset directory relative to the root of the cpuset file system.
.PP
Also the \fI/proc/<pid>/status\fP file for each process has four added
lines, displaying the process's \fICpus_allowed\fP (on which CPUs it may be
scheduled) and \fIMems_allowed\fP (on which memory nodes it may obtain memory),
in the two formats \fBMask Format\fP and \fBList Format\fP (see below)  as shown
in the following example:
.PP
.RS
.nf
Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list:     0\-127
Mems_allowed:   ffffffff,ffffffff
Mems_allowed_list:     0\-63
.fi
.RE
.PP
.\" ================== EXTENDED CAPABILITIES ==================
The "allowed" fields were added in Linux 2.6.24; the "allowed_list" fields
were added in Linux 2.6.26.
.SH "EXTENDED CAPABILITIES"
.\" ================== Exclusive Cpusets ==================
In addition to controlling which \fIcpus\fP and \fImems\fP a process is allowed to
use, cpusets provide the following extended capabilities.
.SS "Exclusive Cpusets"
If a cpuset is marked \fIcpu_exclusive\fP or \fImem_exclusive\fP, no other cpuset,
other than a direct ancestor or descendant, may share any of the same CPUs
or memory nodes.
.PP
.\" ================== Hardwall ==================
A cpuset that is \fImem_exclusive\fP restricts kernel allocations for buffer
cache pages and other internal kernel data pages commonly shared by the
kernel across multiple users.  All cpusets, whether \fImem_exclusive\fP or not,
restrict allocations of memory for user space.  This enables configuring a
system so that several independent jobs can share common kernel data, while
isolating each job's user allocation in its own cpuset.  To do this,
construct a large \fImem_exclusive\fP cpuset to hold all the jobs, and
construct child, non\-\fImem_exclusive\fP cpusets for each individual job.  Only
a small amount of kernel memory, such as requests from interrupt handlers,
is allowed to be placed on memory nodes outside even a \fImem_exclusive\fP
cpuset.
.SS Hardwall
A cpuset that has \fImem_exclusive\fP or \fImem_hardwall\fP set is a \fIhardwall\fP
cpuset.  A \fIhardwall\fP cpuset restricts kernel allocations for page, buffer,
and other data commonly shared by the kernel across multiple users.  All
cpusets, whether \fIhardwall\fP or not, restrict allocations of memory for user
space.
.PP
This enables configuring a system so that several independent jobs can share
common kernel data, such as file system pages, while isolating each job's
user allocation in its own cpuset.  To do this, construct a large
\fIhardwall\fP cpuset to hold all the jobs, and construct child cpusets for
each individual job which are not \fIhardwall\fP cpusets.
.PP
.\" ================== Notify On Release ==================
Only a small amount of kernel memory, such as requests from interrupt
handlers, is allowed to be taken outside even a \fIhardwall\fP cpuset.
.SS "Notify On Release"
If the \fInotify_on_release\fP flag is enabled (1) in a cpuset, then whenever
the last process in the cpuset leaves (exits or attaches to some other
cpuset)  and the last child cpuset of that cpuset is removed, the kernel
will run the command \fI/sbin/cpuset_release_agent\fP, supplying the pathname
(relative to the mount point of the cpuset file system) of the abandoned
cpuset.  This enables automatic removal of abandoned cpusets.
.PP
The default value of \fInotify_on_release\fP in the root cpuset at system boot
is disabled (0).  The default value of other cpusets at creation is the
current value of their parent's \fInotify_on_release\fP setting.
.PP
The command \fI/sbin/cpuset_release_agent\fP is invoked, with the name
(\fI/dev/cpuset\fP relative path)  of the to\-be\-released cpuset in \fIargv[1]\fP.
.PP
The usual contents of the command \fI/sbin/cpuset_release_agent\fP is simply
the shell script:
.in +4n
.nf

#!/bin/sh
rmdir /dev/cpuset/$1
.fi
.in
.PP
.\" ================== Memory Pressure ==================
As with other flag values below, this flag can be changed by writing an
ASCII number 0 or 1 (with optional trailing newline)  into the file, to
clear or set the flag, respectively.
.SS "Memory Pressure"
The \fImemory_pressure\fP of a cpuset provides a simple per\-cpuset running
average of the rate that the processes in a cpuset are attempting to free up
in\-use memory on the nodes of the cpuset to satisfy additional memory
requests.
.PP
This enables batch managers that are monitoring jobs running in dedicated
cpusets to efficiently detect what level of memory pressure that job is
causing.
.PP
This is useful both on tightly managed systems running a wide mix of
submitted jobs, which may choose to terminate or reprioritize jobs that are
trying to use more memory than allowed on the nodes assigned them, and with
tightly coupled, long\-running, massively parallel scientific computing jobs
that will dramatically fail to meet required performance goals if they start
to use more memory than allowed to them.
.PP
This mechanism provides a very economical way for the batch manager to
monitor a cpuset for signs of memory pressure.  It's up to the batch manager
or other user code to decide what action to take if it detects signs of
memory pressure.
.PP
Unless memory pressure calculation is enabled by setting the pseudo\-file
\fI/dev/cpuset/memory_pressure_enabled\fP, it is not computed for any cpuset,
and reads from any \fImemory_pressure\fP always return zero, as represented by
the ASCII string "0\en".  See the \fBWARNINGS\fP section, below.
.PP
A per\-cpuset, running average is employed for the following reasons:
.IP * 3
Because this meter is per\-cpuset rather than per\-process or per virtual
memory region, the system load imposed by a batch scheduler monitoring this
metric is sharply reduced on large systems, because a scan of the tasklist
can be avoided on each set of queries.
.IP *
Because this meter is a running average rather than an accumulating counter,
a batch scheduler can detect memory pressure with a single read, instead of
having to read and accumulate results for a period of time.
.IP *
Because this meter is per\-cpuset rather than per\-process, the batch
scheduler can obtain the key information\(emmemory pressure in a
cpuset\(emwith a single read, rather than having to query and accumulate
results over all the (dynamically changing)  set of processes in the cpuset.
.PP
The \fImemory_pressure\fP of a cpuset is calculated using a per\-cpuset simple
digital filter that is kept within the kernel.  For each cpuset, this filter
tracks the recent rate at which processes attached to that cpuset enter the
kernel direct reclaim code.
.PP
The kernel direct reclaim code is entered whenever a process has to satisfy
a memory page request by first finding some other page to repurpose, due to
lack of any readily available already free pages.  Dirty file system pages
are repurposed by first writing them to disk.  Unmodified file system buffer
pages are repurposed by simply dropping them, though if that page is needed
again, it will have to be reread from disk.
.PP
.\" ================== Memory Spread ==================
The \fImemory_pressure\fP file provides an integer number representing the
recent (half\-life of 10 seconds) rate of entries to the direct reclaim code
caused by any process in the cpuset, in units of reclaims attempted per
second, times 1000.
.SS "Memory Spread"
There are two Boolean flag files per cpuset that control where the kernel
allocates pages for the file\-system buffers and related in\-kernel data
structures.  They are called \fImemory_spread_page\fP and
\fImemory_spread_slab\fP.
.PP
If the per\-cpuset Boolean flag file \fImemory_spread_page\fP is set, then the
kernel will spread the file\-system buffers (page cache) evenly over all the
nodes that the faulting process is allowed to use, instead of preferring to
put those pages on the node where the process is running.
.PP
If the per\-cpuset Boolean flag file \fImemory_spread_slab\fP is set, then the
kernel will spread some file\-system\-related slab caches, such as those for
inodes and directory entries, evenly over all the nodes that the faulting
process is allowed to use, instead of preferring to put those pages on the
node where the process is running.
.PP
The setting of these flags does not affect the data segment (see \fBbrk\fP(2))
or stack segment pages of a process.
.PP
By default, both kinds of memory spreading are off and the kernel prefers to
allocate memory pages on the node local to where the requesting process is
running.  If that node is not allowed by the process's NUMA memory policy or
cpuset configuration or if there are insufficient free memory pages on that
node, then the kernel looks for the nearest node that is allowed and has
sufficient free memory.
.PP
When new cpusets are created, they inherit the memory spread settings of
their parent.
.PP
Setting memory spreading causes allocations for the affected page or slab
caches to ignore the process's NUMA memory policy and be spread instead.
However, the effect of these changes in memory placement caused by
cpuset\-specified memory spreading is hidden from the \fBmbind\fP(2)  or
\fBset_mempolicy\fP(2)  calls.  These two NUMA memory policy calls always
appear to behave as if no cpuset\-specified memory spreading is in effect,
even if it is.  If cpuset memory spreading is subsequently turned off, the
NUMA memory policy most recently specified by these calls is automatically
reapplied.
.PP
Both \fImemory_spread_page\fP and \fImemory_spread_slab\fP are Boolean flag
files.  By default they contain "0", meaning that the feature is off for
that cpuset.  If a "1" is written to that file, that turns the named feature
on.
.PP
Cpuset\-specified memory spreading behaves similarly to what is known (in
other contexts) as round\-robin or interleave memory placement.
.PP
Cpuset\-specified memory spreading can provide substantial performance
improvements for jobs that:
.IP a) 3
need to place thread\-local data on memory nodes close to the CPUs which are
running the threads that most frequently access that data; but also
.IP b)
need to access large file\-system data sets that must to be spread across the
several nodes in the job's cpuset in order to fit.
.PP
.\" ================== Memory Migration ==================
Without this policy, the memory allocation across the nodes in the job's
cpuset can become very uneven, especially for jobs that might have just a
single thread initializing or reading in the data set.
.SS "Memory Migration"
Normally, under the default setting (disabled) of \fImemory_migrate\fP, once a
page is allocated (given a physical page of main memory) then that page
stays on whatever node it was allocated, so long as it remains allocated,
even if the cpuset's memory\-placement policy \fImems\fP subsequently changes.
.PP
When memory migration is enabled in a cpuset, if the \fImems\fP setting of the
cpuset is changed, then any memory page in use by any process in the cpuset
that is on a memory node that is no longer allowed will be migrated to a
memory node that is allowed.
.PP
Furthermore, if a process is moved into a cpuset with \fImemory_migrate\fP
enabled, any memory pages it uses that were on memory nodes allowed in its
previous cpuset, but which are not allowed in its new cpuset, will be
migrated to a memory node allowed in the new cpuset.
.PP
.\" ================== Scheduler Load Balancing ==================
The relative placement of a migrated page within the cpuset is preserved
during these migration operations if possible.  For example, if the page was
on the second valid node of the prior cpuset, then the page will be placed
on the second valid node of the new cpuset, if possible.
.SS "Scheduler Load Balancing"
The kernel scheduler automatically load balances processes.  If one CPU is
underutilized, the kernel will look for processes on other more overloaded
CPUs and move those processes to the underutilized CPU, within the
constraints of such placement mechanisms as cpusets and
\fBsched_setaffinity\fP(2).
.PP
The algorithmic cost of load balancing and its impact on key shared kernel
data structures such as the process list increases more than linearly with
the number of CPUs being balanced.  For example, it costs more to load
balance across one large set of CPUs than it does to balance across two
smaller sets of CPUs, each of half the size of the larger set.  (The precise
relationship between the number of CPUs being balanced and the cost of load
balancing depends on implementation details of the kernel process scheduler,
which is subject to change over time, as improved kernel scheduler
algorithms are implemented.)
.PP
The per\-cpuset flag \fIsched_load_balance\fP provides a mechanism to suppress
this automatic scheduler load balancing in cases where it is not needed and
suppressing it would have worthwhile performance benefits.
.PP
By default, load balancing is done across all CPUs, except those marked
isolated using the kernel boot time "isolcpus=" argument.  (See \fBScheduler
Relax Domain Level\fP, below, to change this default.)
.PP
This default load balancing across all CPUs is not well suited to the
following two situations:
.IP * 3
On large systems, load balancing across many CPUs is expensive.  If the
system is managed using cpusets to place independent jobs on separate sets
of CPUs, full load balancing is unnecessary.
.IP *
Systems supporting real\-time on some CPUs need to minimize system overhead
on those CPUs, including avoiding process load balancing if that is not
needed.
.PP
When the per\-cpuset flag \fIsched_load_balance\fP is enabled (the default
setting), it requests load balancing across all the CPUs in that cpuset's
allowed CPUs, ensuring that load balancing can move a process (not otherwise
pinned, as by \fBsched_setaffinity\fP(2))  from any CPU in that cpuset to any
other.
.PP
When the per\-cpuset flag \fIsched_load_balance\fP is disabled, then the
scheduler will avoid load balancing across the CPUs in that cpuset,
\fIexcept\fP in so far as is necessary because some overlapping cpuset has
\fIsched_load_balance\fP enabled.
.PP
So, for example, if the top cpuset has the flag \fIsched_load_balance\fP
enabled, then the scheduler will load balance across all CPUs, and the
setting of the \fIsched_load_balance\fP flag in other cpusets has no effect, as
we're already fully load balancing.
.PP
Therefore in the above two situations, the flag \fIsched_load_balance\fP should
be disabled in the top cpuset, and only some of the smaller, child cpusets
would have this flag enabled.
.PP
When doing this, you don't usually want to leave any unpinned processes in
the top cpuset that might use nontrivial amounts of CPU, as such processes
may be artificially constrained to some subset of CPUs, depending on the
particulars of this flag setting in descendant cpusets.  Even if such a
process could use spare CPU cycles in some other CPUs, the kernel scheduler
might not consider the possibility of load balancing that process to the
underused CPU.
.PP
.\" ================== Scheduler Relax Domain Level ==================
Of course, processes pinned to a particular CPU can be left in a cpuset that
disables \fIsched_load_balance\fP as those processes aren't going anywhere else
anyway.
.SS "Scheduler Relax Domain Level"
The kernel scheduler performs immediate load balancing whenever a CPU
becomes free or another task becomes runnable.  This load balancing works to
ensure that as many CPUs as possible are usefully employed running tasks.
The kernel also performs periodic load balancing off the software clock
described in \fItime\fP(7).  The setting of \fIsched_relax_domain_level\fP only
applies to immediate load balancing.  Regardless of the
\fIsched_relax_domain_level\fP setting, periodic load balancing is attempted
over all CPUs (unless disabled by turning off \fIsched_load_balance\fP.)  In
any case, of course, tasks will only be scheduled to run on CPUs allowed by
their cpuset, as modified by \fBsched_setaffinity\fP(2)  system calls.
.PP
On small systems, such as those with just a few CPUs, immediate load
balancing is useful to improve system interactivity and to minimize wasteful
idle CPU cycles.  But on large systems, attempting immediate load balancing
across a large number of CPUs can be more costly than it is worth, depending
on the particular performance characteristics of the job mix and the
hardware.
.PP
The exact meaning of the small integer values of \fIsched_relax_domain_level\fP
will depend on internal implementation details of the kernel scheduler code
and on the non\-uniform architecture of the hardware.  Both of these will
evolve over time and vary by system architecture and kernel version.
.PP
As of this writing, when this capability was introduced in Linux 2.6.26, on
certain popular architectures, the positive values of
\fIsched_relax_domain_level\fP have the following meanings.
.sp
.PD 0
.IP \fB(1)\fP 4
Perform immediate load balancing across Hyper\-Thread siblings on the same
core.
.IP \fB(2)\fP
Perform immediate load balancing across other cores in the same package.
.IP \fB(3)\fP
Perform immediate load balancing across other CPUs on the same node or
blade.
.IP \fB(4)\fP
Perform immediate load balancing across over several (implementation detail)
nodes [On NUMA systems].
.IP \fB(5)\fP
Perform immediate load balancing across over all CPUs in system [On NUMA
systems].
.PD
.PP
The \fIsched_relax_domain_level\fP value of zero (0) always means don't perform
immediate load balancing, hence that load balancing is only done
periodically, not immediately when a CPU becomes available or another task
becomes runnable.
.PP
The \fIsched_relax_domain_level\fP value of minus one (\-1)  always means use
the system default value.  The system default value can vary by architecture
and kernel version.  This system default value can be changed by kernel
boot\-time "relax_domain_level=" argument.
.PP
In the case of multiple overlapping cpusets which have conflicting
\fIsched_relax_domain_level\fP values, then the highest such value applies to
all CPUs in any of the overlapping cpusets.  In such cases, the value
\fBminus one (\-1)\fP is the lowest value, overridden by any other value, and
the value \fBzero (0)\fP is the next lowest value.
.SH FORMATS
.\" ================== Mask Format ==================
The following formats are used to represent sets of CPUs and memory nodes.
.SS "Mask Format"
The \fBMask Format\fP is used to represent CPU and memory\-node bitmasks in the
\fI/proc/<pid>/status\fP file.
.PP
This format displays each 32\-bit word in hexadecimal (using ASCII characters
"0" \- "9" and "a" \- "f"); words are filled with leading zeros, if required.
For masks longer than one word, a comma separator is used between words.
Words are displayed in big\-endian order, which has the most significant bit
first.  The hex digits within a word are also in big\-endian order.
.PP
The number of 32\-bit words displayed is the minimum number needed to display
all bits of the bitmask, based on the size of the bitmask.
.PP
Examples of the \fBMask Format\fP:
.PP
.RS
.nf
00000001                        # just bit 0 set
40000000,00000000,00000000      # just bit 94 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32\-39 set
00000000,000E3862               # 1,5,6,11\-13,17\-19 set
.fi
.RE
.PP
A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
.PP
.RS
.nf
00000001,00000001,00010117
.fi
.RE
.PP
.\" ================== List Format ==================
The first "1" is for bit 64, the second for bit 32, the third for bit 16,
the fourth for bit 8, the fifth for bit 4, and the "7" is for bits 2, 1, and
0.
.SS "List Format"
The \fBList Format\fP for \fIcpus\fP and \fImems\fP is a comma\-separated list of CPU
or memory\-node numbers and ranges of numbers, in ASCII decimal.
.PP
Examples of the \fBList Format\fP:
.PP
.RS
.nf
0\-4,9           # bits 0, 1, 2, 3, 4, and 9 set
0\-2,7,12\-14     # bits 0, 1, 2, 7, 12, 13, and 14 set
.fi
.RE
.\" ================== RULES ==================
.SH RULES
The following rules apply to each cpuset:
.IP * 3
Its CPUs and memory nodes must be a (possibly equal)  subset of its
parent's.
.IP *
It can only be marked \fIcpu_exclusive\fP if its parent is.
.IP *
It can only be marked \fImem_exclusive\fP if its parent is.
.IP *
If it is \fIcpu_exclusive\fP, its CPUs may not overlap any sibling.
.IP *
.\" ================== PERMISSIONS ==================
If it is \fImemory_exclusive\fP, its memory nodes may not overlap any sibling.
.SH PERMISSIONS
The permissions of a cpuset are determined by the permissions of the
directories and pseudo\-files in the cpuset file system, normally mounted at
\fI/dev/cpuset\fP.
.PP
For instance, a process can put itself in some other cpuset (than its
current one) if it can write the \fItasks\fP file for that cpuset.  This
requires execute permission on the encompassing directories and write
permission on the \fItasks\fP file.
.PP
An additional constraint is applied to requests to place some other process
in a cpuset.  One process may not attach another to a cpuset unless it would
have permission to send that process a signal (see \fBkill\fP(2)).
.PP
A process may create a child cpuset if it can access and write the parent
cpuset directory.  It can modify the CPUs or memory nodes in a cpuset if it
can access that cpuset's directory (execute permissions on the each of the
parent directories) and write the corresponding \fIcpus\fP or \fImems\fP file.
.PP
There is one minor difference between the manner in which these permissions
are evaluated and the manner in which normal file\-system operation
permissions are evaluated.  The kernel interprets relative pathnames
starting at a process's current working directory.  Even if one is operating
on a cpuset file, relative pathnames are interpreted relative to the
process's current working directory, not relative to the process's current
cpuset.  The only ways that cpuset paths relative to a process's current
cpuset can be used are if either the process's current working directory is
its cpuset (it first did a \fBcd\fP or \fBchdir\fP(2)  to its cpuset directory
beneath \fI/dev/cpuset\fP, which is a bit unusual)  or if some user code
converts the relative cpuset path to a full file\-system path.
.PP
.\" ================== WARNINGS ==================
In theory, this means that user code should specify cpusets using absolute
pathnames, which requires knowing the mount point of the cpuset file system
(usually, but not necessarily, \fI/dev/cpuset\fP).  In practice, all user level
code that this author is aware of simply assumes that if the cpuset file
system is mounted, then it is mounted at \fI/dev/cpuset\fP.  Furthermore, it is
common practice for carefully written user code to verify the presence of
the pseudo\-file \fI/dev/cpuset/tasks\fP in order to verify that the cpuset
pseudo\-file system is currently mounted.
.SH WARNINGS
.SS "Enabling memory_pressure"
By default, the per\-cpuset file \fImemory_pressure\fP always contains zero
(0).  Unless this feature is enabled by writing "1" to the pseudo\-file
\fI/dev/cpuset/memory_pressure_enabled\fP, the kernel does not compute
per\-cpuset \fImemory_pressure\fP.
.SS "Using the echo command"
.\" Gack!  csh(1)'s echo does this
When using the \fBecho\fP command at the shell prompt to change the values of
cpuset files, beware that the built\-in \fBecho\fP command in some shells does
not display an error message if the \fBwrite\fP(2)  system call fails.  For
example, if the command:
.in +4n
.nf

echo 19 > mems

.fi
.in
failed because memory node 19 was not allowed (perhaps the current system
does not have a memory node 19), then the \fBecho\fP command might not display
any error.  It is better to use the \fB/bin/echo\fP external command to change
cpuset file settings, as this command will display \fBwrite\fP(2)  errors, as
in the example:
.in +4n
.nf

/bin/echo 19 > mems
/bin/echo: write error: Invalid argument
.fi
.in
.\" ================== EXCEPTIONS ==================
.SH EXCEPTIONS
.SS "Memory placement"
Not all allocations of system memory are constrained by cpusets, for the
following reasons.
.PP
If hot\-plug functionality is used to remove all the CPUs that are currently
assigned to a cpuset, then the kernel will automatically update the
\fIcpus_allowed\fP of all processes attached to CPUs in that cpuset to allow
all CPUs.  When memory hot\-plug functionality for removing memory nodes is
available, a similar exception is expected to apply there as well.  In
general, the kernel prefers to violate cpuset placement, rather than
starving a process that has had all its allowed CPUs or memory nodes taken
offline.  User code should reconfigure cpusets to only refer to online CPUs
and memory nodes when using hot\-plug to add or remove such resources.
.PP
A few kernel\-critical, internal memory\-allocation requests, marked
GFP_ATOMIC, must be satisfied immediately.  The kernel may drop some request
or malfunction if one of these allocations fail.  If such a request cannot
be satisfied within the current process's cpuset, then we relax the cpuset,
and look for memory anywhere we can find it.  It's better to violate the
cpuset than stress the kernel.
.PP
Allocations of memory requested by kernel drivers while processing an
interrupt lack any relevant process context, and are not confined by
cpusets.
.SS "Renaming cpusets"
.\" ================== ERRORS ==================
You can use the \fBrename\fP(2)  system call to rename cpusets.  Only simple
renaming is supported; that is, changing the name of a cpuset directory is
permitted, but moving a directory into a different directory is not
permitted.
.SH エラー
The Linux kernel implementation of cpusets sets \fIerrno\fP to specify the
reason for a failed system call affecting cpusets.
.PP
The possible \fIerrno\fP settings and their meaning when set on a failed cpuset
call are as listed below.
.TP 
\fBE2BIG\fP
Attempted a \fBwrite\fP(2)  on a special cpuset file with a length larger than
some kernel\-determined upper limit on the length of such writes.
.TP 
\fBEACCES\fP
Attempted to \fBwrite\fP(2)  the process ID (PID) of a process to a cpuset
\fItasks\fP file when one lacks permission to move that process.
.TP 
\fBEACCES\fP
Attempted to add, using \fBwrite\fP(2), a CPU or memory node to a cpuset, when
that CPU or memory node was not already in its parent.
.TP 
\fBEACCES\fP
Attempted to set, using \fBwrite\fP(2), \fIcpu_exclusive\fP or \fImem_exclusive\fP on
a cpuset whose parent lacks the same setting.
.TP 
\fBEACCES\fP
Attempted to \fBwrite\fP(2)  a \fImemory_pressure\fP file.
.TP 
\fBEACCES\fP
Attempted to create a file in a cpuset directory.
.TP 
\fBEBUSY\fP
Attempted to remove, using \fBrmdir\fP(2), a cpuset with attached processes.
.TP 
\fBEBUSY\fP
Attempted to remove, using \fBrmdir\fP(2), a cpuset with child cpusets.
.TP 
\fBEBUSY\fP
Attempted to remove a CPU or memory node from a cpuset that is also in a
child of that cpuset.
.TP 
\fBEEXIST\fP
Attempted to create, using \fBmkdir\fP(2), a cpuset that already exists.
.TP 
\fBEEXIST\fP
Attempted to \fBrename\fP(2)  a cpuset to a name that already exists.
.TP 
\fBEFAULT\fP
Attempted to \fBread\fP(2)  or \fBwrite\fP(2)  a cpuset file using a buffer that
is outside the writing processes accessible address space.
.TP 
\fBEINVAL\fP
Attempted to change a cpuset, using \fBwrite\fP(2), in a way that would violate
a \fIcpu_exclusive\fP or \fImem_exclusive\fP attribute of that cpuset or any of
its siblings.
.TP 
\fBEINVAL\fP
Attempted to \fBwrite\fP(2)  an empty \fIcpus\fP or \fImems\fP list to a cpuset which
has attached processes or child cpusets.
.TP 
\fBEINVAL\fP
Attempted to \fBwrite\fP(2)  a \fIcpus\fP or \fImems\fP list which included a range
with the second number smaller than the first number.
.TP 
\fBEINVAL\fP
Attempted to \fBwrite\fP(2)  a \fIcpus\fP or \fImems\fP list which included an
invalid character in the string.
.TP 
\fBEINVAL\fP
Attempted to \fBwrite\fP(2)  a list to a \fIcpus\fP file that did not include any
online CPUs.
.TP 
\fBEINVAL\fP
Attempted to \fBwrite\fP(2)  a list to a \fImems\fP file that did not include any
online memory nodes.
.TP 
\fBEINVAL\fP
Attempted to \fBwrite\fP(2)  a list to a \fImems\fP file that included a node that
held no memory.
.TP 
\fBEIO\fP
Attempted to \fBwrite\fP(2)  a string to a cpuset \fItasks\fP file that does not
begin with an ASCII decimal integer.
.TP 
\fBEIO\fP
Attempted to \fBrename\fP(2)  a cpuset into a different directory.
.TP 
\fBENAMETOOLONG\fP
Attempted to \fBread\fP(2)  a \fI/proc/<pid>/cpuset\fP file for a cpuset
path that is longer than the kernel page size.
.TP 
\fBENAMETOOLONG\fP
Attempted to create, using \fBmkdir\fP(2), a cpuset whose base directory name
is longer than 255 characters.
.TP 
\fBENAMETOOLONG\fP
Attempted to create, using \fBmkdir\fP(2), a cpuset whose full pathname,
including the mount point (typically "/dev/cpuset/") prefix, is longer than
4095 characters.
.TP 
\fBENODEV\fP
The cpuset was removed by another process at the same time as a \fBwrite\fP(2)
was attempted on one of the pseudo\-files in the cpuset directory.
.TP 
\fBENOENT\fP
Attempted to create, using \fBmkdir\fP(2), a cpuset in a parent cpuset that
doesn't exist.
.TP 
\fBENOENT\fP
Attempted to \fBaccess\fP(2)  or \fBopen\fP(2)  a nonexistent file in a cpuset
directory.
.TP 
\fBENOMEM\fP
Insufficient memory is available within the kernel; can occur on a variety
of system calls affecting cpusets, but only if the system is extremely short
of memory.
.TP 
\fBENOSPC\fP
Attempted to \fBwrite\fP(2)  the process ID (PID)  of a process to a cpuset
\fItasks\fP file when the cpuset had an empty \fIcpus\fP or empty \fImems\fP setting.
.TP 
\fBENOSPC\fP
Attempted to \fBwrite\fP(2)  an empty \fIcpus\fP or \fImems\fP setting to a cpuset
that has tasks attached.
.TP 
\fBENOTDIR\fP
Attempted to \fBrename\fP(2)  a nonexistent cpuset.
.TP 
\fBEPERM\fP
Attempted to remove a file from a cpuset directory.
.TP 
\fBERANGE\fP
Specified a \fIcpus\fP or \fImems\fP list to the kernel which included a number
too large for the kernel to set in its bitmasks.
.TP 
\fBESRCH\fP
.\" ================== VERSIONS ==================
Attempted to \fBwrite\fP(2)  the process ID (PID) of a nonexistent process to a
cpuset \fItasks\fP file.
.SH バージョン
.\" ================== NOTES ==================
Cpusets appeared in version 2.6.12 of the Linux kernel.
.SH 注意
.\" ================== BUGS ==================
Despite its name, the \fIpid\fP parameter is actually a thread ID, and each
thread in a threaded group can be attached to a different cpuset.  The value
returned from a call to \fBgettid\fP(2)  can be passed in the argument \fIpid\fP.
.SH バグ
.\" ================== EXAMPLE ==================
\fImemory_pressure\fP cpuset files can be opened for writing, creation, or
truncation, but then the \fBwrite\fP(2)  fails with \fIerrno\fP set to \fBEACCES\fP,
and the creation and truncation options on \fBopen\fP(2)  have no effect.
.SH EXAMPLE
The following examples demonstrate querying and setting cpuset options using
shell commands.
.SS "Creating and attaching to a cpuset."
To create a new cpuset and attach the current command shell to it, the steps
are:
.sp
.PD 0
.IP 1) 4
mkdir /dev/cpuset (if not already done)
.IP 2)
mount \-t cpuset none /dev/cpuset (if not already done)
.IP 3)
Create the new cpuset using \fBmkdir\fP(1).
.IP 4)
Assign CPUs and memory nodes to the new cpuset.
.IP 5)
Attach the shell to the new cpuset.
.PD
.PP
For example, the following sequence of commands will set up a cpuset named
"Charlie", containing just CPUs 2 and 3, and memory node 1, and then attach
the current shell to that cpuset.
.in +4n
.nf

$\fB mkdir /dev/cpuset\fP
$\fB mount \-t cpuset cpuset /dev/cpuset\fP
$\fB cd /dev/cpuset\fP
$\fB mkdir Charlie\fP
$\fB cd Charlie\fP
$\fB /bin/echo 2\-3 > cpus\fP
$\fB /bin/echo 1 > mems\fP
$\fB /bin/echo $$ > tasks\fP
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
$\fB cat /proc/self/cpuset\fP
.fi
.in
.SS "Migrating a job to different memory nodes."
To migrate a job (the set of processes attached to a cpuset)  to different
CPUs and memory nodes in the system, including moving the memory pages
currently allocated to that job, perform the following steps.
.sp
.PD 0
.IP 1) 4
Let's say we want to move the job in cpuset \fIalpha\fP (CPUs 4\-7 and memory
nodes 2\-3) to a new cpuset \fIbeta\fP (CPUs 16\-19 and memory nodes 8\-9).
.IP 2)
First create the new cpuset \fIbeta\fP.
.IP 3)
Then allow CPUs 16\-19 and memory nodes 8\-9 in \fIbeta\fP.
.IP 4)
Then enable \fImemory_migration\fP in \fIbeta\fP.
.IP 5)
Then move each process from \fIalpha\fP to \fIbeta\fP.
.PD
.PP
The following sequence of commands accomplishes this.
.in +4n
.nf

$\fB cd /dev/cpuset\fP
$\fB mkdir beta\fP
$\fB cd beta\fP
$\fB /bin/echo 16\-19 > cpus\fP
$\fB /bin/echo 8\-9 > mems\fP
$\fB /bin/echo 1 > memory_migrate\fP
$\fB while read i; do /bin/echo $i; done < ../alpha/tasks > tasks\fP
.fi
.in
.PP
The above should move any processes in \fIalpha\fP to \fIbeta\fP, and any memory
held by these processes on memory nodes 2\-3 to memory nodes 8\-9,
respectively.
.PP
Notice that the last step of the above sequence did not do:
.in +4n
.nf

$\fB cp ../alpha/tasks tasks\fP
.fi
.in
.PP
The \fIwhile\fP loop, rather than the seemingly easier use of the \fBcp\fP(1)
command, was necessary because only one process PID at a time may be written
to the \fItasks\fP file.
.PP
The same effect (writing one PID at a time) as the \fIwhile\fP loop can be
accomplished more efficiently, in fewer keystrokes and in syntax that works
on any shell, but alas more obscurely, by using the \fB\-u\fP (unbuffered)
option of \fBsed\fP(1):
.in +4n

.nf
$\fB sed \-un p < ../alpha/tasks > tasks\fP
.fi
.in
.\" ================== SEE ALSO ==================
.SH 関連項目
\fBtaskset\fP(1), \fBget_mempolicy\fP(2), \fBgetcpu\fP(2), \fBmbind\fP(2),
\fBsched_getaffinity\fP(2), \fBsched_setaffinity\fP(2), \fBsched_setscheduler\fP(2),
\fBset_mempolicy\fP(2), \fBCPU_SET\fP(3), \fBproc\fP(5), \fBnuma\fP(7),
\fBmigratepages\fP(8), \fBnumactl\fP(8)
.PP
The kernel source file \fIDocumentation/cpusets.txt\fP.