1 .\" Copyright (c) 2008 Silicon Graphics, Inc.
3 .\" Author: Paul Jackson (http://oss.sgi.com/projects/cpusets)
5 .\" This is free documentation; you can redistribute it and/or
6 .\" modify it under the terms of the GNU General Public License
7 .\" version 2 as published by the Free Software Foundation.
9 .\" The GNU General Public License's references to "object code"
10 .\" and "executables" are to be interpreted as the output of any
11 .\" document formatting or typesetting system, including
12 .\" intermediate and printed output.
14 .\" This manual is distributed in the hope that it will be useful,
15 .\" but WITHOUT ANY WARRANTY; without even the implied warranty of
16 .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
17 .\" GNU General Public License for more details.
19 .\" You should have received a copy of the GNU General Public
20 .\" License along with this manual; if not, write to the Free
21 .\" Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
24 .TH CPUSET 7 2008-11-12 "Linux" "Linux Programmer's Manual"
26 cpuset \- confine processes to processor and memory node subsets
28 The cpuset file system is a pseudo-file-system interface
29 to the kernel cpuset mechanism,
30 which is used to control the processor placement
31 and memory placement of processes.
32 It is commonly mounted at
35 On systems with kernels compiled with built in support for cpusets,
36 all processes are attached to a cpuset, and cpusets are always present.
37 If a system supports cpusets, then it will have the entry
40 .IR /proc/filesystems .
41 By mounting the cpuset file system (see the
44 the administrator can configure the cpusets on a system
45 to control the processor and memory placement of processes
47 By default, if the cpuset configuration
48 on a system is not modified or if the cpuset file
49 system is not even mounted, then the cpuset mechanism,
50 though present, has no affect on the system's behavior.
52 A cpuset defines a list of CPUs and memory nodes.
54 The CPUs of a system include all the logical processing
55 units on which a process can execute, including, if present,
56 multiple processor cores within a package and Hyper-Threads
57 within a processor core.
58 Memory nodes include all distinct
59 banks of main memory; small and SMP systems typically have
60 just one memory node that contains all the system's main memory,
61 while NUMA (non-uniform memory access) systems have multiple memory nodes.
63 Cpusets are represented as directories in a hierarchical
64 pseudo-file system, where the top directory in the hierarchy
66 represents the entire system (all online CPUs and memory nodes)
67 and any cpuset that is the child (descendant) of
68 another parent cpuset contains a subset of that parent's
69 CPUs and memory nodes.
70 The directories and files representing cpusets have normal
71 file-system permissions.
73 Every process in the system belongs to exactly one cpuset.
74 A process is confined to only run on the CPUs in
75 the cpuset it belongs to, and to allocate memory only
76 on the memory nodes in that cpuset.
79 the child process is placed in the same cpuset as its parent.
80 With sufficient privilege, a process may be moved from one
81 cpuset to another and the allowed CPUs and memory nodes
82 of an existing cpuset may be changed.
84 When the system begins booting, a single cpuset is
85 defined that includes all CPUs and memory nodes on the
86 system, and all processes are in that cpuset.
87 During the boot process, or later during normal system operation,
88 other cpusets may be created, as subdirectories of this top cpuset,
89 under the control of the system administrator,
90 and processes may be placed in these other cpusets.
92 Cpusets are integrated with the
93 .BR sched_setaffinity (2)
94 scheduling affinity mechanism and the
98 memory-placement mechanisms in the kernel.
99 Neither of these mechanisms let a process make use
100 of a CPU or memory node that is not allowed by that process's cpuset.
101 If changes to a process's cpuset placement conflict with these
102 other mechanisms, then cpuset placement is enforced
103 even if it means overriding these other mechanisms.
104 The kernel accomplishes this overriding by silently
105 restricting the CPUs and memory nodes requested by
106 these other mechanisms to those allowed by the
107 invoking process's cpuset.
108 This can result in these
109 other calls returning an error, if for example, such
110 a call ends up requesting an empty set of CPUs or
111 memory nodes, after that request is restricted to
112 the invoking process's cpuset.
114 Typically, a cpuset is used to manage
115 the CPU and memory-node confinement for a set of
116 cooperating processes such as a batch scheduler job, and these
117 other mechanisms are used to manage the placement of
118 individual processes or memory regions within that set or job.
122 represents a cpuset and contains a fixed set of pseudo-files
123 describing the state of that cpuset.
125 New cpusets are created using the
130 The properties of a cpuset, such as its flags, allowed
131 CPUs and memory nodes, and attached processes, are queried and modified
132 by reading or writing to the appropriate file in that cpuset's directory,
135 The pseudo-files in each cpuset directory are automatically created when
136 the cpuset is created, as a result of the
139 It is not possible to directly add or remove these pseudo-files.
141 A cpuset directory that contains no child cpuset directories,
142 and has no attached processes, can be removed using
146 It is not necessary, or possible,
147 to remove the pseudo-files inside the directory before removing it.
149 The pseudo-files in each cpuset directory are
150 small text files that may be read and
151 written using traditional shell utilities such as
155 or from a program by using file I/O library functions or system calls,
163 The pseudo-files in a cpuset directory represent internal kernel
164 state and do not have any persistent image on disk.
165 Each of these per-cpuset files is listed and described below.
166 .\" ====================== tasks ======================
169 List of the process IDs (PIDs) of the processes in that cpuset.
170 The list is formatted as a series of ASCII
171 decimal numbers, each followed by a newline.
172 A process may be added to a cpuset (automatically removing
173 it from the cpuset that previously contained it) by writing its
176 file (with or without a trailing newline.)
179 only one PID may be written to the
182 If a string is written that contains more
183 than one PID, only the first one will be used.
184 .\" =================== notify_on_release ===================
188 If set (1), that cpuset will receive special handling
189 after it is released, that is, after all processes cease using
190 it (i.e., terminate or are moved to a different cpuset)
191 and all child cpuset directories have been removed.
192 See the \fBNotify On Release\fR section, below.
193 .\" ====================== cpus ======================
196 List of the physical numbers of the CPUs on which processes
197 in that cpuset are allowed to execute.
198 See \fBList Format\fR below for a description of the
202 The CPUs allowed to a cpuset may be changed by
203 writing a new list to its
206 .\" ==================== cpu_exclusive ====================
210 If set (1), the cpuset has exclusive use of
211 its CPUs (no sibling or cousin cpuset may overlap CPUs).
212 By default this is off (0).
213 Newly created cpusets also initially default this to off (0).
217 cpusets if they share the same parent cpuset in the
222 cpusets if neither is the ancestor of the other.
225 setting, if one cpuset is the ancestor of another,
226 and if both of these cpusets have nonempty
230 must overlap, because the
232 of any cpuset are always a subset of the
234 of its parent cpuset.
235 .\" ====================== mems ======================
238 List of memory nodes on which processes in this cpuset are
239 allowed to allocate memory.
240 See \fBList Format\fR below for a description of the
243 .\" ==================== mem_exclusive ====================
247 If set (1), the cpuset has exclusive use of
248 its memory nodes (no sibling or cousin may overlap).
249 Also if set (1), the cpuset is a \fBHardwall\fR cpuset (see below.)
250 By default this is off (0).
251 Newly created cpusets also initially default this to off (0).
255 setting, if one cpuset is the ancestor of another,
256 then their memory nodes must overlap, because the memory
257 nodes of any cpuset are always a subset of the memory nodes
258 of that cpuset's parent cpuset.
259 .\" ==================== mem_hardwall ====================
261 .IR mem_hardwall " (since Linux 2.6.26)"
263 If set (1), the cpuset is a \fBHardwall\fR cpuset (see below.)
264 Unlike \fBmem_exclusive\fR,
265 there is no constraint on whether cpusets
266 marked \fBmem_hardwall\fR may have overlapping
267 memory nodes with sibling or cousin cpusets.
268 By default this is off (0).
269 Newly created cpusets also initially default this to off (0).
270 .\" ==================== memory_migrate ====================
272 .IR memory_migrate " (since Linux 2.6.16)"
274 If set (1), then memory migration is enabled.
275 By default this is off (0).
276 See the \fBMemory Migration\fR section, below.
277 .\" ==================== memory_pressure ====================
279 .IR memory_pressure " (since Linux 2.6.16)"
280 A measure of how much memory pressure the processes in this
282 See the \fBMemory Pressure\fR section, below.
284 .I memory_pressure_enabled
285 is enabled, always has value zero (0).
286 This file is read-only.
290 .\" ================= memory_pressure_enabled =================
292 .IR memory_pressure_enabled " (since Linux 2.6.16)"
294 This file is only present in the root cpuset, normally
298 calculations are enabled for all cpusets in the system.
299 By default this is off (0).
301 \fBMemory Pressure\fR section, below.
302 .\" ================== memory_spread_page ==================
304 .IR memory_spread_page " (since Linux 2.6.17)"
306 If set (1), pages in the kernel page cache
307 (file-system buffers) are uniformly spread across the cpuset.
308 By default this is off (0) in the top cpuset,
309 and inherited from the parent cpuset in
310 newly created cpusets.
311 See the \fBMemory Spread\fR section, below.
312 .\" ================== memory_spread_slab ==================
314 .IR memory_spread_slab " (since Linux 2.6.17)"
316 If set (1), the kernel slab caches
317 for file I/O (directory and inode structures) are
318 uniformly spread across the cpuset.
319 By default this is off (0) in the top cpuset,
320 and inherited from the parent cpuset in
321 newly created cpusets.
322 See the \fBMemory Spread\fR section, below.
323 .\" ================== sched_load_balance ==================
325 .IR sched_load_balance " (since Linux 2.6.24)"
327 If set (1, the default) the kernel will
328 automatically load balance processes in that cpuset over
329 the allowed CPUs in that cpuset.
331 kernel will avoid load balancing processes in this cpuset,
333 some other cpuset with overlapping CPUs has its
334 .I sched_load_balance
336 See \fBScheduler Load Balancing\fR, below, for further details.
337 .\" ================== sched_relax_domain_level ==================
339 .IR sched_relax_domain_level " (since Linux 2.6.26)"
340 Integer, between \-1 and a small positive value.
342 .I sched_relax_domain_level
343 controls the width of the range of CPUs over which the kernel scheduler
344 performs immediate rebalancing of runnable tasks across CPUs.
346 .I sched_load_balance
347 is disabled, then the setting of
348 .I sched_relax_domain_level
349 does not matter, as no such load balancing is done.
351 .I sched_load_balance
352 is enabled, then the higher the value of the
353 .IR sched_relax_domain_level ,
355 the range of CPUs over which immediate load balancing is attempted.
356 See \fBScheduler Relax Domain Level\fR, below, for further details.
357 .\" ================== proc cpuset ==================
359 In addition to the above pseudo-files in each directory below
361 each process has a pseudo-file,
362 .IR /proc/<pid>/cpuset ,
363 that displays the path of the process's cpuset directory
364 relative to the root of the cpuset file system.
365 .\" ================== proc status ==================
368 .I /proc/<pid>/status
369 file for each process has four added lines,
370 displaying the process's
372 (on which CPUs it may be scheduled) and
374 (on which memory nodes it may obtain memory),
375 in the two formats \fBMask Format\fR and \fBList Format\fR (see below)
376 as shown in the following example:
380 Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
381 Cpus_allowed_list: 0-127
382 Mems_allowed: ffffffff,ffffffff
383 Mems_allowed_list: 0-63
387 The "allowed" fields were added in Linux 2.6.24;
388 the "allowed_list" fields were added in Linux 2.6.26.
389 .\" ================== EXTENDED CAPABILITIES ==================
390 .SH EXTENDED CAPABILITIES
391 In addition to controlling which
395 a process is allowed to use, cpusets provide the following
396 extended capabilities.
397 .\" ================== Exclusive Cpusets ==================
398 .SS Exclusive Cpusets
399 If a cpuset is marked
403 no other cpuset, other than a direct ancestor or descendant,
404 may share any of the same CPUs or memory nodes.
408 restricts kernel allocations for
409 buffer cache pages and other internal kernel data pages
410 commonly shared by the kernel across
414 or not, restrict allocations of memory for user space.
415 This enables configuring a
416 system so that several independent jobs can share common kernel data,
417 while isolating each job's user allocation in
419 To do this, construct a large
421 cpuset to hold all the jobs, and construct child,
422 .RI non- mem_exclusive
423 cpusets for each individual job.
424 Only a small amount of kernel memory,
425 such as requests from interrupt handlers, is allowed to be
426 placed on memory nodes
430 .\" ================== Hardwall ==================
441 cpuset restricts kernel allocations for page, buffer,
442 and other data commonly shared by the kernel across multiple users.
445 or not, restrict allocations of memory for user space.
447 This enables configuring a system so that several independent
448 jobs can share common kernel data, such as file system pages,
449 while isolating each job's user allocation in its own cpuset.
450 To do this, construct a large
453 all the jobs, and construct child cpusets for each individual
458 Only a small amount of kernel memory, such as requests from
459 interrupt handlers, is allowed to be taken outside even a
462 .\" ================== Notify On Release ==================
463 .SS Notify On Release
466 flag is enabled (1) in a cpuset,
467 then whenever the last process in the cpuset leaves
468 (exits or attaches to some other cpuset)
469 and the last child cpuset of that cpuset is removed,
470 the kernel will run the command
471 .IR /sbin/cpuset_release_agent ,
472 supplying the pathname (relative to the mount point of the
473 cpuset file system) of the abandoned cpuset.
474 This enables automatic removal of abandoned cpusets.
478 in the root cpuset at system boot is disabled (0).
479 The default value of other cpusets at creation
480 is the current value of their parent's
485 .I /sbin/cpuset_release_agent
486 is invoked, with the name
489 of the to-be-released cpuset in
492 The usual contents of the command
493 .I /sbin/cpuset_release_agent
494 is simply the shell script:
503 As with other flag values below, this flag can
504 be changed by writing an ASCII
505 number 0 or 1 (with optional trailing newline)
506 into the file, to clear or set the flag, respectively.
507 .\" ================== Memory Pressure ==================
511 of a cpuset provides a simple per-cpuset running average of
512 the rate that the processes in a cpuset are attempting to free up in-use
513 memory on the nodes of the cpuset to satisfy additional memory requests.
515 This enables batch managers that are monitoring jobs running in dedicated
516 cpusets to efficiently detect what level of memory pressure that job
519 This is useful both on tightly managed systems running a wide mix of
520 submitted jobs, which may choose to terminate or reprioritize jobs that
521 are trying to use more memory than allowed on the nodes assigned them,
522 and with tightly coupled, long-running, massively parallel scientific
523 computing jobs that will dramatically fail to meet required performance
524 goals if they start to use more memory than allowed to them.
526 This mechanism provides a very economical way for the batch manager
527 to monitor a cpuset for signs of memory pressure.
528 It's up to the batch manager or other user code to decide
529 what action to take if it detects signs of memory pressure.
531 Unless memory pressure calculation is enabled by setting the pseudo-file
532 .IR /dev/cpuset/memory_pressure_enabled ,
533 it is not computed for any cpuset, and reads from any
535 always return zero, as represented by the ASCII string "0\en".
536 See the \fBWARNINGS\fR section, below.
538 A per-cpuset, running average is employed for the following reasons:
540 Because this meter is per-cpuset rather than per-process or per virtual
541 memory region, the system load imposed by a batch scheduler monitoring
542 this metric is sharply reduced on large systems, because a scan of
543 the tasklist can be avoided on each set of queries.
545 Because this meter is a running average rather than an accumulating
546 counter, a batch scheduler can detect memory pressure with a
547 single read, instead of having to read and accumulate results
548 for a period of time.
550 Because this meter is per-cpuset rather than per-process,
551 the batch scheduler can obtain the key information\(emmemory
552 pressure in a cpuset\(emwith a single read, rather than having to
553 query and accumulate results over all the (dynamically changing)
554 set of processes in the cpuset.
558 of a cpuset is calculated using a per-cpuset simple digital filter
559 that is kept within the kernel.
560 For each cpuset, this filter tracks
561 the recent rate at which processes attached to that cpuset enter the
562 kernel direct reclaim code.
564 The kernel direct reclaim code is entered whenever a process has to
565 satisfy a memory page request by first finding some other page to
566 repurpose, due to lack of any readily available already free pages.
567 Dirty file system pages are repurposed by first writing them
569 Unmodified file system buffer pages are repurposed
570 by simply dropping them, though if that page is needed again, it
571 will have to be reread from disk.
575 file provides an integer number representing the recent (half-life of
576 10 seconds) rate of entries to the direct reclaim code caused by any
577 process in the cpuset, in units of reclaims attempted per second,
579 .\" ================== Memory Spread ==================
581 There are two Boolean flag files per cpuset that control where the
582 kernel allocates pages for the file-system buffers and related
583 in-kernel data structures.
585 .I memory_spread_page
587 .IR memory_spread_slab .
589 If the per-cpuset Boolean flag file
590 .I memory_spread_page
592 the kernel will spread the file-system buffers (page cache) evenly
593 over all the nodes that the faulting process is allowed to use, instead
594 of preferring to put those pages on the node where the process is running.
596 If the per-cpuset Boolean flag file
597 .I memory_spread_slab
599 then the kernel will spread some file-system-related slab caches,
600 such as those for inodes and directory entries, evenly over all the nodes
601 that the faulting process is allowed to use, instead of preferring to
602 put those pages on the node where the process is running.
604 The setting of these flags does not affect the data segment
607 or stack segment pages of a process.
609 By default, both kinds of memory spreading are off and the kernel
610 prefers to allocate memory pages on the node local to where the
611 requesting process is running.
612 If that node is not allowed by the
613 process's NUMA memory policy or cpuset configuration or if there are
614 insufficient free memory pages on that node, then the kernel looks
615 for the nearest node that is allowed and has sufficient free memory.
617 When new cpusets are created, they inherit the memory spread settings
620 Setting memory spreading causes allocations for the affected page or
621 slab caches to ignore the process's NUMA memory policy and be spread
623 However, the effect of these changes in memory placement
624 caused by cpuset-specified memory spreading is hidden from the
627 .BR set_mempolicy (2)
629 These two NUMA memory policy calls always appear to behave as if
630 no cpuset-specified memory spreading is in effect, even if it is.
631 If cpuset memory spreading is subsequently turned off, the NUMA
632 memory policy most recently specified by these calls is automatically
636 .I memory_spread_page
638 .I memory_spread_slab
639 are Boolean flag files.
640 By default they contain "0", meaning that the feature is off
642 If a "1" is written to that file, that turns the named feature on.
644 Cpuset-specified memory spreading behaves similarly to what is known
645 (in other contexts) as round-robin or interleave memory placement.
647 Cpuset-specified memory spreading can provide substantial performance
648 improvements for jobs that:
650 need to place thread-local data on
651 memory nodes close to the CPUs which are running the threads that most
652 frequently access that data; but also
654 need to access large file-system data sets that must to be spread
655 across the several nodes in the job's cpuset in order to fit.
658 the memory allocation across the nodes in the job's cpuset
659 can become very uneven,
660 especially for jobs that might have just a single
661 thread initializing or reading in the data set.
662 .\" ================== Memory Migration ==================
664 Normally, under the default setting (disabled) of
666 once a page is allocated (given a physical page
667 of main memory) then that page stays on whatever node it
668 was allocated, so long as it remains allocated, even if the
669 cpuset's memory-placement policy
671 subsequently changes.
673 When memory migration is enabled in a cpuset, if the
675 setting of the cpuset is changed, then any memory page in use by any
676 process in the cpuset that is on a memory node that is no longer
677 allowed will be migrated to a memory node that is allowed.
679 Furthermore, if a process is moved into a cpuset with
681 enabled, any memory pages it uses that were on memory nodes allowed
682 in its previous cpuset, but which are not allowed in its new cpuset,
683 will be migrated to a memory node allowed in the new cpuset.
685 The relative placement of a migrated page within
686 the cpuset is preserved during these migration operations if possible.
688 if the page was on the second valid node of the prior cpuset,
689 then the page will be placed on the second valid node of the new cpuset,
691 .\" ================== Scheduler Load Balancing ==================
692 .SS Scheduler Load Balancing
693 The kernel scheduler automatically load balances processes.
694 If one CPU is underutilized,
695 the kernel will look for processes on other more
696 overloaded CPUs and move those processes to the underutilized CPU,
697 within the constraints of such placement mechanisms as cpusets and
698 .BR sched_setaffinity (2).
700 The algorithmic cost of load balancing and its impact on key shared
701 kernel data structures such as the process list increases more than
702 linearly with the number of CPUs being balanced.
704 costs more to load balance across one large set of CPUs than it does
705 to balance across two smaller sets of CPUs, each of half the size
707 (The precise relationship between the number of CPUs being balanced
708 and the cost of load balancing depends
709 on implementation details of the kernel process scheduler, which is
710 subject to change over time, as improved kernel scheduler algorithms
714 .I sched_load_balance
715 provides a mechanism to suppress this automatic scheduler load
716 balancing in cases where it is not needed and suppressing it would have
717 worthwhile performance benefits.
719 By default, load balancing is done across all CPUs, except those
720 marked isolated using the kernel boot time "isolcpus=" argument.
721 (See \fBScheduler Relax Domain Level\fR, below, to change this default.)
723 This default load balancing across all CPUs is not well suited to
724 the following two situations:
726 On large systems, load balancing across many CPUs is expensive.
727 If the system is managed using cpusets to place independent jobs
728 on separate sets of CPUs, full load balancing is unnecessary.
730 Systems supporting real-time on some CPUs need to minimize
731 system overhead on those CPUs, including avoiding process load
732 balancing if that is not needed.
734 When the per-cpuset flag
735 .I sched_load_balance
736 is enabled (the default setting),
737 it requests load balancing across
738 all the CPUs in that cpuset's allowed CPUs,
739 ensuring that load balancing can move a process (not otherwise pinned,
741 .BR sched_setaffinity (2))
742 from any CPU in that cpuset to any other.
744 When the per-cpuset flag
745 .I sched_load_balance
746 is disabled, then the
747 scheduler will avoid load balancing across the CPUs in that cpuset,
748 \fIexcept\fR in so far as is necessary because some overlapping cpuset
750 .I sched_load_balance
753 So, for example, if the top cpuset has the flag
754 .I sched_load_balance
755 enabled, then the scheduler will load balance across all
756 CPUs, and the setting of the
757 .I sched_load_balance
758 flag in other cpusets has no effect,
759 as we're already fully load balancing.
761 Therefore in the above two situations, the flag
762 .I sched_load_balance
763 should be disabled in the top cpuset, and only some of the smaller,
764 child cpusets would have this flag enabled.
766 When doing this, you don't usually want to leave any unpinned processes in
767 the top cpuset that might use nontrivial amounts of CPU, as such processes
768 may be artificially constrained to some subset of CPUs, depending on
769 the particulars of this flag setting in descendant cpusets.
770 Even if such a process could use spare CPU cycles in some other CPUs,
771 the kernel scheduler might not consider the possibility of
772 load balancing that process to the underused CPU.
774 Of course, processes pinned to a particular CPU can be left in a cpuset
776 .I sched_load_balance
777 as those processes aren't going anywhere else anyway.
778 .\" ================== Scheduler Relax Domain Level ==================
779 .SS Scheduler Relax Domain Level
780 The kernel scheduler performs immediate load balancing whenever
781 a CPU becomes free or another task becomes runnable.
783 balancing works to ensure that as many CPUs as possible are usefully
784 employed running tasks.
785 The kernel also performs periodic load
786 balancing off the software clock described in
789 .I sched_relax_domain_level
790 only applies to immediate load balancing.
792 .I sched_relax_domain_level
793 setting, periodic load balancing is attempted over all CPUs
794 (unless disabled by turning off
795 .IR sched_load_balance .)
796 In any case, of course, tasks will only be scheduled to run on
797 CPUs allowed by their cpuset, as modified by
798 .BR sched_setaffinity (2)
801 On small systems, such as those with just a few CPUs, immediate load
802 balancing is useful to improve system interactivity and to minimize
803 wasteful idle CPU cycles.
804 But on large systems, attempting immediate
805 load balancing across a large number of CPUs can be more costly than
806 it is worth, depending on the particular performance characteristics
807 of the job mix and the hardware.
809 The exact meaning of the small integer values of
810 .I sched_relax_domain_level
811 will depend on internal
812 implementation details of the kernel scheduler code and on the
813 non-uniform architecture of the hardware.
814 Both of these will evolve
815 over time and vary by system architecture and kernel version.
817 As of this writing, when this capability was introduced in Linux
818 2.6.26, on certain popular architectures, the positive values of
819 .I sched_relax_domain_level
820 have the following meanings.
824 Perform immediate load balancing across Hyper-Thread
825 siblings on the same core.
827 Perform immediate load balancing across other cores in the same package.
829 Perform immediate load balancing across other CPUs
830 on the same node or blade.
832 Perform immediate load balancing across over several
833 (implementation detail) nodes [On NUMA systems].
835 Perform immediate load balancing across over all CPUs
836 in system [On NUMA systems].
840 .I sched_relax_domain_level
841 value of zero (0) always means
842 don't perform immediate load balancing,
843 hence that load balancing is only done periodically,
844 not immediately when a CPU becomes available or another task becomes
848 .I sched_relax_domain_level
849 value of minus one (\-1)
850 always means use the system default value.
851 The system default value can vary by architecture and kernel version.
852 This system default value can be changed by kernel
853 boot-time "relax_domain_level=" argument.
855 In the case of multiple overlapping cpusets which have conflicting
856 .I sched_relax_domain_level
857 values, then the highest such value
858 applies to all CPUs in any of the overlapping cpusets.
860 the value \fBminus one (\-1)\fR is the lowest value, overridden by any
861 other value, and the value \fBzero (0)\fR is the next lowest value.
863 The following formats are used to represent sets of
864 CPUs and memory nodes.
865 .\" ================== Mask Format ==================
867 The \fBMask Format\fR is used to represent CPU and memory-node bitmasks
869 .I /proc/<pid>/status
872 This format displays each 32-bit
873 word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f");
874 words are filled with leading zeros, if required.
875 For masks longer than one word, a comma separator is used between words.
876 Words are displayed in big-endian
877 order, which has the most significant bit first.
878 The hex digits within a word are also in big-endian order.
880 The number of 32-bit words displayed is the minimum number needed to
881 display all bits of the bitmask, based on the size of the bitmask.
883 Examples of the \fBMask Format\fR:
887 00000001 # just bit 0 set
888 40000000,00000000,00000000 # just bit 94 set
889 00000001,00000000,00000000 # just bit 64 set
890 000000ff,00000000 # bits 32-39 set
891 00000000,000E3862 # 1,5,6,11-13,17-19 set
895 A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
899 00000001,00000001,00010117
903 The first "1" is for bit 64, the
904 second for bit 32, the third for bit 16, the fourth for bit 8, the
905 fifth for bit 4, and the "7" is for bits 2, 1, and 0.
906 .\" ================== List Format ==================
908 The \fBList Format\fR for
912 is a comma-separated list of CPU or memory-node
913 numbers and ranges of numbers, in ASCII decimal.
915 Examples of the \fBList Format\fR:
919 0-4,9 # bits 0, 1, 2, 3, 4, and 9 set
920 0-2,7,12-14 # bits 0, 1, 2, 7, 12, 13, and 14 set
923 .\" ================== RULES ==================
925 The following rules apply to each cpuset:
927 Its CPUs and memory nodes must be a (possibly equal)
928 subset of its parent's.
930 It can only be marked
934 It can only be marked
940 its CPUs may not overlap any sibling.
943 .IR memory_exclusive ,
944 its memory nodes may not overlap any sibling.
945 .\" ================== PERMISSIONS ==================
947 The permissions of a cpuset are determined by the permissions
948 of the directories and pseudo-files in the cpuset file system,
952 For instance, a process can put itself in some other cpuset (than
953 its current one) if it can write the
955 file for that cpuset.
956 This requires execute permission on the encompassing directories
957 and write permission on the
961 An additional constraint is applied to requests to place some
962 other process in a cpuset.
963 One process may not attach another to
964 a cpuset unless it would have permission to send that process
968 A process may create a child cpuset if it can access and write the
969 parent cpuset directory.
970 It can modify the CPUs or memory nodes
971 in a cpuset if it can access that cpuset's directory (execute
972 permissions on the each of the parent directories) and write the
979 There is one minor difference between the manner in which these
980 permissions are evaluated and the manner in which normal file-system
981 operation permissions are evaluated.
982 The kernel interprets
983 relative pathnames starting at a process's current working directory.
984 Even if one is operating on a cpuset file, relative pathnames
985 are interpreted relative to the process's current working directory,
986 not relative to the process's current cpuset.
988 cpuset paths relative to a process's current cpuset can be used are
989 if either the process's current working directory is its cpuset
994 to its cpuset directory beneath
996 which is a bit unusual)
997 or if some user code converts the relative cpuset path to a
998 full file-system path.
1000 In theory, this means that user code should specify cpusets
1001 using absolute pathnames, which requires knowing the mount point of
1002 the cpuset file system (usually, but not necessarily,
1004 In practice, all user level code that this author is aware of
1005 simply assumes that if the cpuset file system is mounted, then
1008 Furthermore, it is common practice for carefully written
1009 user code to verify the presence of the pseudo-file
1010 .I /dev/cpuset/tasks
1011 in order to verify that the cpuset pseudo-file system
1012 is currently mounted.
1013 .\" ================== WARNINGS ==================
1015 .SS Enabling memory_pressure
1016 By default, the per-cpuset file
1018 always contains zero (0).
1019 Unless this feature is enabled by writing "1" to the pseudo-file
1020 .IR /dev/cpuset/memory_pressure_enabled ,
1022 not compute per-cpuset
1023 .IR memory_pressure .
1024 .SS Using the echo command
1027 command at the shell prompt to change the values of cpuset files,
1028 beware that the built-in
1030 command in some shells does not display an error message if the
1033 .\" Gack! csh(1)'s echo does this
1034 For example, if the command:
1042 failed because memory node 19 was not allowed (perhaps
1043 the current system does not have a memory node 19), then the
1045 command might not display any error.
1046 It is better to use the
1048 external command to change cpuset file settings, as this
1049 command will display
1051 errors, as in the example:
1056 /bin/echo: write error: Invalid argument
1059 .\" ================== EXCEPTIONS ==================
1061 .SS Memory placement
1062 Not all allocations of system memory are constrained by cpusets,
1063 for the following reasons.
1065 If hot-plug functionality is used to remove all the CPUs that are
1066 currently assigned to a cpuset, then the kernel will automatically
1069 of all processes attached to CPUs in that cpuset
1071 When memory hot-plug functionality for removing
1072 memory nodes is available, a similar exception is expected to apply
1074 In general, the kernel prefers to violate cpuset placement,
1075 rather than starving a process that has had all its allowed CPUs or
1076 memory nodes taken offline.
1077 User code should reconfigure cpusets to only refer to online CPUs
1078 and memory nodes when using hot-plug to add or remove such resources.
1080 A few kernel-critical, internal memory-allocation requests, marked
1081 GFP_ATOMIC, must be satisfied immediately.
1082 The kernel may drop some
1083 request or malfunction if one of these allocations fail.
1084 If such a request cannot be satisfied within the current process's cpuset,
1085 then we relax the cpuset, and look for memory anywhere we can find it.
1086 It's better to violate the cpuset than stress the kernel.
1088 Allocations of memory requested by kernel drivers while processing
1089 an interrupt lack any relevant process context, and are not confined
1091 .SS Renaming cpusets
1094 system call to rename cpusets.
1095 Only simple renaming is supported; that is, changing the name of a cpuset
1096 directory is permitted, but moving a directory into
1097 a different directory is not permitted.
1098 .\" ================== ERRORS ==================
1100 The Linux kernel implementation of cpusets sets
1102 to specify the reason for a failed system call affecting cpusets.
1106 settings and their meaning when set on
1107 a failed cpuset call are as listed below.
1112 on a special cpuset file
1113 with a length larger than some kernel-determined upper
1114 limit on the length of such writes.
1119 the process ID (PID) of a process to a cpuset
1121 file when one lacks permission to move that process.
1124 Attempted to add, using
1126 a CPU or memory node to a cpuset, when that CPU or memory node was
1127 not already in its parent.
1130 Attempted to set, using
1135 on a cpuset whose parent lacks the same setting.
1145 Attempted to create a file in a cpuset directory.
1148 Attempted to remove, using
1150 a cpuset with attached processes.
1153 Attempted to remove, using
1155 a cpuset with child cpusets.
1159 a CPU or memory node from a cpuset
1160 that is also in a child of that cpuset.
1163 Attempted to create, using
1165 a cpuset that already exists.
1170 a cpuset to a name that already exists.
1178 a buffer that is outside the writing processes accessible address space.
1181 Attempted to change a cpuset, using
1183 in a way that would violate a
1187 attribute of that cpuset or any of its siblings.
1196 list to a cpuset which has attached processes or child cpusets.
1205 list which included a range with the second number smaller than
1215 list which included an invalid character in the string.
1222 file that did not include any online CPUs.
1229 file that did not include any online memory nodes.
1236 file that included a node that held no memory.
1241 a string to a cpuset
1244 does not begin with an ASCII decimal integer.
1249 a cpuset into a different directory.
1255 .I /proc/<pid>/cpuset
1256 file for a cpuset path that is longer than the kernel page size.
1259 Attempted to create, using
1261 a cpuset whose base directory name is longer than 255 characters.
1264 Attempted to create, using
1266 a cpuset whose full pathname,
1267 including the mount point (typically "/dev/cpuset/") prefix,
1268 is longer than 4095 characters.
1271 The cpuset was removed by another process at the same time as a
1273 was attempted on one of the pseudo-files in the cpuset directory.
1276 Attempted to create, using
1278 a cpuset in a parent cpuset that doesn't exist.
1285 a nonexistent file in a cpuset directory.
1288 Insufficient memory is available within the kernel; can occur
1289 on a variety of system calls affecting cpusets, but only if the
1290 system is extremely short of memory.
1295 the process ID (PID)
1296 of a process to a cpuset
1298 file when the cpuset had an empty
1311 setting to a cpuset that
1317 a nonexistent cpuset.
1320 Attempted to remove a file from a cpuset directory.
1327 list to the kernel which included a number too large for the kernel
1328 to set in its bitmasks.
1333 the process ID (PID) of a nonexistent process to a cpuset
1336 .\" ================== VERSIONS ==================
1338 Cpusets appeared in version 2.6.12 of the Linux kernel.
1339 .\" ================== NOTES ==================
1341 Despite its name, the
1343 parameter is actually a thread ID,
1344 and each thread in a threaded group can be attached to a different
1346 The value returned from a call to
1348 can be passed in the argument
1350 .\" ================== BUGS ==================
1353 cpuset files can be opened
1354 for writing, creation, or truncation, but then the
1360 and the creation and truncation options on
1363 .\" ================== EXAMPLE ==================
1365 The following examples demonstrate querying and setting cpuset
1366 options using shell commands.
1367 .SS Creating and attaching to a cpuset.
1368 To create a new cpuset and attach the current command shell to it,
1373 mkdir /dev/cpuset (if not already done)
1375 mount \-t cpuset none /dev/cpuset (if not already done)
1377 Create the new cpuset using
1380 Assign CPUs and memory nodes to the new cpuset.
1382 Attach the shell to the new cpuset.
1385 For example, the following sequence of commands will set up a cpuset
1386 named "Charlie", containing just CPUs 2 and 3, and memory node 1,
1387 and then attach the current shell to that cpuset.
1391 .RB "$" " mkdir /dev/cpuset"
1392 .RB "$" " mount \-t cpuset cpuset /dev/cpuset"
1393 .RB "$" " cd /dev/cpuset"
1394 .RB "$" " mkdir Charlie"
1395 .RB "$" " cd Charlie"
1396 .RB "$" " /bin/echo 2-3 > cpus"
1397 .RB "$" " /bin/echo 1 > mems"
1398 .RB "$" " /bin/echo $$ > tasks"
1399 # The current shell is now running in cpuset Charlie
1400 # The next line should display '/Charlie'
1401 .RB "$" " cat /proc/self/cpuset"
1404 .SS Migrating a job to different memory nodes.
1405 To migrate a job (the set of processes attached to a cpuset)
1406 to different CPUs and memory nodes in the system, including moving
1407 the memory pages currently allocated to that job,
1408 perform the following steps.
1412 Let's say we want to move the job in cpuset
1414 (CPUs 4-7 and memory nodes 2-3) to a new cpuset
1416 (CPUs 16-19 and memory nodes 8-9).
1418 First create the new cpuset
1421 Then allow CPUs 16-19 and memory nodes 8-9 in
1429 Then move each process from
1435 The following sequence of commands accomplishes this.
1439 .RB "$" " cd /dev/cpuset"
1440 .RB "$" " mkdir beta"
1442 .RB "$" " /bin/echo 16-19 > cpus"
1443 .RB "$" " /bin/echo 8-9 > mems"
1444 .RB "$" " /bin/echo 1 > memory_migrate"
1445 .RB "$" " while read i; do /bin/echo $i; done < ../alpha/tasks > tasks"
1449 The above should move any processes in
1453 and any memory held by these processes on memory nodes 2-3 to memory
1454 nodes 8-9, respectively.
1456 Notice that the last step of the above sequence did not do:
1460 .RB "$" " cp ../alpha/tasks tasks"
1466 loop, rather than the seemingly easier use of the
1468 command, was necessary because
1469 only one process PID at a time may be written to the
1473 The same effect (writing one PID at a time) as the
1475 loop can be accomplished more efficiently, in fewer keystrokes and in
1476 syntax that works on any shell, but alas more obscurely, by using the
1478 (unbuffered) option of
1483 .RB "$" " sed \-un p < ../alpha/tasks > tasks"
1486 .\" ================== SEE ALSO ==================
1489 .BR get_mempolicy (2),
1492 .BR sched_getaffinity (2),
1493 .BR sched_setaffinity (2),
1494 .BR sched_setscheduler (2),
1495 .BR set_mempolicy (2),
1499 .BR migratepages (8),
1502 The kernel source file
1503 .IR Documentation/cpusets.txt .