+==========
+MD Cluster
+==========
+
The cluster MD is a shared-device RAID for a cluster, it supports
two levels: raid1 and raid10 (limited support).
1. On-disk format
+=================
Separate write-intent-bitmaps are used for each cluster node.
The bitmaps record all writes that may have been started on that node,
-and may not yet have finished. The on-disk layout is:
+and may not yet have finished. The on-disk layout is::
-0 4k 8k 12k
--------------------------------------------------------------------
-| idle | md super | bm super [0] + bits |
-| bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
-| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
-| bm bits [3, contd] | | |
+ 0 4k 8k 12k
+ -------------------------------------------------------------------
+ | idle | md super | bm super [0] + bits |
+ | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
+ | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
+ | bm bits [3, contd] | | |
During "normal" functioning we assume the filesystem ensures that only
one node writes to any given block at a time, so a write request will
2. DLM Locks for management
+===========================
There are three groups of locks for managing the device:
2.1 Bitmap lock resource (bm_lockres)
+-------------------------------------
The bm_lockres protects individual node bitmaps. They are named in
the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
joins the cluster.
2.2 Message passing locks
+-------------------------
Each node has to communicate with other nodes when starting or ending
resync, and for metadata superblock updates. This communication is
with the Lock Value Block (LVB) of one of the "message" lock.
2.3 new-device management
+-------------------------
A single lock: "no-new-dev" is used to co-ordinate the addition of
new devices - this must be synchronized across the array.
Normally all nodes hold a concurrent-read lock on this device.
3. Communication
+================
Messages can be broadcast to all nodes, and the sender waits for all
other nodes to acknowledge the message before proceeding. Only one
message can be processed at a time.
3.1 Message Types
+-----------------
There are six types of messages which are passed:
- 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has
+3.1.1 METADATA_UPDATED
+^^^^^^^^^^^^^^^^^^^^^^
+
+ informs other nodes that the metadata has
been updated, and the node must re-read the md superblock. This is
performed synchronously. It is primarily used to signal device
failure.
- 3.1.2 RESYNCING: informs other nodes that a resync is initiated or
+3.1.2 RESYNCING
+^^^^^^^^^^^^^^^
+ informs other nodes that a resync is initiated or
ended so that each node may suspend or resume the region. Each
RESYNCING message identifies a range of the devices that the
sending node is about to resync. This overrides any previous
notification from that node: only one ranged can be resynced at a
time per-node.
- 3.1.3 NEWDISK: informs other nodes that a device is being added to
+3.1.3 NEWDISK
+^^^^^^^^^^^^^
+
+ informs other nodes that a device is being added to
the array. Message contains an identifier for that device. See
below for further details.
- 3.1.4 REMOVE: A failed or spare device is being removed from the
+3.1.4 REMOVE
+^^^^^^^^^^^^
+
+ A failed or spare device is being removed from the
array. The slot-number of the device is included in the message.
- 3.1.5 RE_ADD: A failed device is being re-activated - the assumption
+ 3.1.5 RE_ADD:
+
+ A failed device is being re-activated - the assumption
is that it has been determined to be working again.
- 3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap
+ 3.1.6 BITMAP_NEEDS_SYNC:
+
+ If a node is stopped locally but the bitmap
isn't clean, then another node is informed to take the ownership of
resync.
3.2 Communication mechanism
+---------------------------
The DLM LVB is used to communicate within nodes of the cluster. There
are three resources used for the purpose:
- 3.2.1 token: The resource which protects the entire communication
+3.2.1 token
+^^^^^^^^^^^
+ The resource which protects the entire communication
system. The node having the token resource is allowed to
communicate.
- 3.2.2 message: The lock resource which carries the data to
- communicate.
+3.2.2 message
+^^^^^^^^^^^^^
+ The lock resource which carries the data to communicate.
- 3.2.3 ack: The resource, acquiring which means the message has been
+3.2.3 ack
+^^^^^^^^^
+
+ The resource, acquiring which means the message has been
acknowledged by all nodes in the cluster. The BAST of the resource
is used to inform the receiving node that a node wants to
communicate.
The algorithm is:
- 1. receive status - all nodes have concurrent-reader lock on "ack".
+ 1. receive status - all nodes have concurrent-reader lock on "ack"::
+
+ sender receiver receiver
+ "ack":CR "ack":CR "ack":CR
- sender receiver receiver
- "ack":CR "ack":CR "ack":CR
+ 2. sender get EX on "token",
+ sender get EX on "message"::
- 2. sender get EX on "token"
- sender get EX on "message"
- sender receiver receiver
- "token":EX "ack":CR "ack":CR
- "message":EX
- "ack":CR
+ sender receiver receiver
+ "token":EX "ack":CR "ack":CR
+ "message":EX
+ "ack":CR
Sender checks that it still needs to send a message. Messages
received or other events that happened while waiting for the
"token" may have made this message inappropriate or redundant.
- 3. sender writes LVB.
+ 3. sender writes LVB
+
sender down-convert "message" from EX to CW
+
sender try to get EX of "ack"
- [ wait until all receivers have *processed* the "message" ]
- [ triggered by bast of "ack" ]
- receiver get CR on "message"
- receiver read LVB
- receiver processes the message
- [ wait finish ]
- receiver releases "ack"
- receiver tries to get PR on "message"
+ ::
+
+ [ wait until all receivers have *processed* the "message" ]
- sender receiver receiver
- "token":EX "message":CR "message":CR
- "message":CW
- "ack":EX
+ [ triggered by bast of "ack" ]
+ receiver get CR on "message"
+ receiver read LVB
+ receiver processes the message
+ [ wait finish ]
+ receiver releases "ack"
+ receiver tries to get PR on "message"
+
+ sender receiver receiver
+ "token":EX "message":CR "message":CR
+ "message":CW
+ "ack":EX
4. triggered by grant of EX on "ack" (indicating all receivers
have processed message)
+
sender down-converts "ack" from EX to CR
+
sender releases "message"
+
sender releases "token"
- receiver upconvert to PR on "message"
- receiver get CR of "ack"
- receiver release "message"
- sender receiver receiver
- "ack":CR "ack":CR "ack":CR
+ ::
+
+ receiver upconvert to PR on "message"
+ receiver get CR of "ack"
+ receiver release "message"
+
+ sender receiver receiver
+ "ack":CR "ack":CR "ack":CR
4. Handling Failures
+====================
4.1 Node Failure
+----------------
When a node fails, the DLM informs the cluster with the slot
number. The node starts a cluster recovery thread. The cluster
- cleans the bitmap of the failed node
- releases bitmap<number> lock of the failed node
- initiates resync of the bitmap on the current node
- md_check_recovery is invoked within recover_bitmaps,
- then md_check_recovery -> metadata_update_start/finish,
- it will lock the communication by lock_comm.
- Which means when one node is resyncing it blocks all
- other nodes from writing anywhere on the array.
+ md_check_recovery is invoked within recover_bitmaps,
+ then md_check_recovery -> metadata_update_start/finish,
+ it will lock the communication by lock_comm.
+ Which means when one node is resyncing it blocks all
+ other nodes from writing anywhere on the array.
The resync process is the regular md resync. However, in a clustered
environment when a resync is performed, it needs to tell other nodes
particular I/O range should be suspended or not.
4.2 Device Failure
+==================
Device failures are handled and communicated with the metadata update
routine. When a node detects a device failure it does not allow
acknowledged by all other nodes.
5. Adding a new Device
+----------------------
For adding a new device, it is necessary that all nodes "see" the new
device to be added. For this, the following algorithm is used:
- 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
+ 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
- 2. Node 1 sends a NEWDISK message with uuid and slot number
- 3. Other nodes issue kobject_uevent_env with uuid and slot number
+ 2. Node 1 sends a NEWDISK message with uuid and slot number
+ 3. Other nodes issue kobject_uevent_env with uuid and slot number
(Steps 4,5 could be a udev rule)
- 4. In userspace, the node searches for the disk, perhaps
+ 4. In userspace, the node searches for the disk, perhaps
using blkid -t SUB_UUID=""
- 5. Other nodes issue either of the following depending on whether
+ 5. Other nodes issue either of the following depending on whether
the disk was found:
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
- disc.number set to slot number)
+ disc.number set to slot number)
ioctl(CLUSTERED_DISK_NACK)
- 6. Other nodes drop lock on "no-new-devs" (CR) if device is found
- 7. Node 1 attempts EX lock on "no-new-dev"
- 8. If node 1 gets the lock, it sends METADATA_UPDATED after
+ 6. Other nodes drop lock on "no-new-devs" (CR) if device is found
+ 7. Node 1 attempts EX lock on "no-new-dev"
+ 8. If node 1 gets the lock, it sends METADATA_UPDATED after
unmarking the disk as SpareLocal
- 9. If not (get "no-new-dev" lock), it fails the operation and sends
+ 9. If not (get "no-new-dev" lock), it fails the operation and sends
METADATA_UPDATED.
10. Other nodes get the information whether a disk is added or not
by the following METADATA_UPDATED.
-6. Module interface.
+6. Module interface
+===================
There are 17 call-backs which the md core can make to the cluster
module. Understanding these can give a good overview of the whole
process.
6.1 join(nodes) and leave()
+---------------------------
These are called when an array is started with a clustered bitmap,
and when the array is stopped. join() ensures the cluster is
Only the first 'nodes' nodes in the cluster can use the array.
6.2 slot_number()
+-----------------
Reports the slot number advised by the cluster infrastructure.
Range is from 0 to nodes-1.
6.3 resync_info_update()
+------------------------
This updates the resync range that is stored in the bitmap lock.
The starting point is updated as the resync progresses. The
It does *not* send a RESYNCING message.
6.4 resync_start(), resync_finish()
+-----------------------------------
These are called when resync/recovery/reshape starts or stops.
They update the resyncing range in the bitmap lock and also
resync_finish() also sends a BITMAP_NEEDS_SYNC message which
allows some other node to take over.
-6.5 metadata_update_start(), metadata_update_finish(),
- metadata_update_cancel().
+6.5 metadata_update_start(), metadata_update_finish(), metadata_update_cancel()
+-------------------------------------------------------------------------------
metadata_update_start is used to get exclusive access to
the metadata. If a change is still needed once that access is
can be used to release the lock.
6.6 area_resyncing()
+--------------------
This combines two elements of functionality.
a node failure.
6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()
+---------------------------------------------------------------
These are used to manage the new-disk protocol described above.
When a new device is added, add_new_disk_start() is called before
new_disk_ack() is called.
6.8 remove_disk()
+-----------------
This is called when a spare or failed device is removed from
the array. It causes a REMOVE message to be send to other nodes.
6.9 gather_bitmaps()
+--------------------
This sends a RE_ADD message to all other nodes and then
gathers bitmap information from all bitmaps. This combined
bitmap is then used to recovery the re-added device.
6.10 lock_all_bitmaps() and unlock_all_bitmaps()
+------------------------------------------------
These are called when change bitmap to none. If a node plans
to clear the cluster raid's bitmap, it need to make sure no other
accordingly.
7. Unsupported features
+=======================
There are somethings which are not supported by cluster MD yet.