Lustre quick cheats MDS - metadata server MDT - metadata target (the disk) OSS - Object storage server

OST - Object storage target (the disk) MGS - management server - holds cluster conguration and exports mount point to clients - lustre clients contact it to mount the le system - can be part of the MDT (commonly done, it is low overhead) - can serve multiple le systems modprobe -v lustre loads lustre insmod | grep lustre shows if the kernel modules are loaded rmmod_lustre removes the lustre kernel modules servers need: - lustre kernel modules - ldiskfs kernel modules - lustre user space "stuff" - a special version of e2fsprogs clients need a kernel that matches the client kernel modules and the base lustre user space rpm User space commands: mkfs.lustre lctl lfs mount.lustre tune2fs tunefs.lustre by default one inode takes up 4k - consider this when sizing the MDT

Things get much more complex if you are using IB rather than FC, in that case you need to get all your IB setup rst. You can use in IB TCP network for lnet trafc, that seems to work well.

============== don't forget to put the /etc/modprobe.d/lustre.conf to have your lnet interface to use on all your machines mkfs examples: MDS: mkfs.lustre --fsname=lustre --mdt --mgs /dev/yourfs (this will include the mgs in the le system) just mount with mount -t lustre /dev/yourfs /your/mountpoint this usually does not get put in the /etc/fstab

OSS: mkfs.lustre --fsname=lustre --ost -mgsnode=IP_of_MDS@tcp /dev/yourdevice (if using ethernet) mount: mount -t lustre /dev/yourdevice /your/mountpoint this usually does not get put in the /etc/fstab client: they usually get theirs in the /etc/fstab: IP_of_MDS@tcp:/lustre /mnt/lustre/yourdir lustre defaults, _netdev 0 0 (often ock is a used option of MPI is involved). if you mount by hand: mount -t lustre IP_of_MDS:/lustre /mnt/lustre/yourdir

======================= /etc/modprobe.d/lustre.conf looks like: options lnet networks="tcp1(eth1)" multiple lnets can be used for load balancing ======================== lfs used to setup striping rst you need to setup your OSTS pools using lctl commands or "o2ib0", etc.

by default stripes in 4MB breadths lfs getstripe and lfs setstripe are the commands used lfs setstripe: --count is number of OST's to stripe across (-1 means use all available) --size is the stripe size in multiples of 64k (default is 1MB) ============= This shows the mounts in the pool: lfs df -h UUID % Mounted on lustre-MDT0000_UUID 0% /lustre[MDT:0] lustre-OST0000_UUID 0% /lustre[OST:0] lustre-OST0001_UUID 0% /lustre[OST:1]

bytes 130.4G 146.7G 146.7G

Used Available 459.6M 1.4G 459.6M 122.5G 137.8G 138.8G

Use

filesystem summary: 0% /lustre

293.3G

1.9G

276.6G

================================= Good discussion stripe alignment: h5p://www.nics.tennessee.edu/io-:ps ======================================== LNLL's I O guide - lots of good info: h5ps://compu:ng.llnl.gov/LCdocs/ioguide/ ========================================= From N ASA good info on striping: h5p://www.nas.nasa.gov/hecc/support/kb/ Lustre-Basics_224.html