初探OCFS2-配置服务

初探OCFS2-配置服务

2016/11/28

目的：研究如何使用 iscsi 存储。一、认识据zstack的童鞋们介绍：OCFS2是主流对接 SAN 云存储的方式存储的选择，通常有以下方式：1、NAS1）NFS简单；自建有可靠性问题，商业的 NFS 产品更靠谱；2）分布式文件系统，以 GlusterFS 为例；易懂；建议使用 glusterfs 的接口来访问，但也支持通过 nfs 来访问；2、IP SAN（iscsi）1）使用iscsiadm 发现一下 target 即可2）研究 ovirt 是如何使用 iscsi 存储的看到一个讨论ovirt通过lvm来使用iscsi的方式http://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi按照字面意思，我的理解（未实验）是：a、每个 host 挂载 iscsi 存储b、通过 lvm 划分出 vm diskc、在运行了 vm 的 host 上 通过 lvchange 来激活指定的 vm disk lvchange -ay vol_group/logical_vol3）通过集群文件系统（cluster filesystem: OCFS2,GFS,etc. ）来控制读写 iscsi 存储这就是本文的初衷，接下来展开记录 OCFS2 是如何使用的。二、配置1、准备安装 oracle uek 内核（因为新版本的 ocfs2 内核模块被 uek 整合了）[root@n149 ~]# wget http://public-yum.oracle.com/public-yum-ol6.repo -O /etc/yum.repos.d/public-yum-ol6.repo[root@n149 ~]# rpm --import http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6[root@n149 ~]# yum install yum-plugin-downloadonly -y[root@n149 ~]# mkdir /tmp/ocfs2 && cd /tmp/ocfs2/[root@n149 ocfs2]# yum install --downloadonly --downloaddir=/tmp/ocfs2/ kernel-uek ocfs2-tools -y2、安装 uek[root@n149 ocfs2]# yum localinstall kernel-uek*使用新的内核，编辑 grub 调整启动顺序：[root@n149 ocfs2]# vim /etc/grub.conf重启系统后查看：[root@n149 ocfs2]# uname -r2.6.39-400.290.1.el6uek.x86_64查看 ocfs2 模块是否加载：[root@n149 ocfs2]# modprobe ocfs2[root@n149 ocfs2]# lsmod |grep ococfs2                1094439  0 ocfs2_nodemanager     223253  1 ocfs2configfs               29308  2 ocfs2_nodemanagerocfs2_stackglue        11896  1 ocfs2jbd2                   93114  2 ocfs2,ext43、安装 ocfs2-tools[root@n149 ocfs2]# yum localinstall ocfs2-tools* -y4、配置 ocfs2 服务[root@n149 ~]# vim /etc/hosts10.50.200.149 n149.ocfs2.test10.50.200.150 n150.ocfs2.test[root@n149 ~]# service o2cb loadchecking debugfs...Mounting configfs filesystem at /sys/kernel/config: OKLoading stack plugin "o2cb": OKLoading filesystem "ocfs2_dlmfs": OKCreating directory '/dlm': OKMounting ocfs2_dlmfs filesystem at /dlm: OK[root@n149 ~]# service o2cb onlinechecking debugfs...Setting cluster stack "o2cb": OKChecking O2CB cluster configuration : Failed符合预期，继续配置集群：[root@n149 ~]# o2cb_ctl -C -n prdcluster -t cluster -a name=prdcluster[root@n149 ~]# o2cb_ctl -C -n n149.ocfs2.test -t node -a number=0 -a ip_address=10.50.200.149 -a ip_port=11111 -a cluster=prdcluster[root@n149 ~]# o2cb_ctl -C -n n150.ocfs2.test -t node -a number=1 -a ip_address=10.50.200.150 -a ip_port=11111 -a cluster=prdcluster[root@n149 ~]# cat /etc/ocfs2/cluster.conf node:        name = n149.ocfs2.test        cluster = prdcluster        number = 0        ip_address = 10.50.200.149        ip_port = 11111node:        name = n150.ocfs2.test        cluster = prdcluster        number = 1        ip_address = 10.50.200.150        ip_port = 11111cluster:        name = prdcluster        heartbeat_mode = local        node_count = 2        [root@n149 ~]# service o2cb configure[root@n149 ~]# service o2cb online上述操作，在每个节点上执行。其他可选操作：[root@n149 ~]# service o2cb status       Driver for "configfs": LoadedFilesystem "configfs": MountedStack glue driver: LoadedStack plugin "o2cb": LoadedDriver for "ocfs2_dlmfs": LoadedFilesystem "ocfs2_dlmfs": MountedChecking O2CB cluster "prdcluster": Online  Heartbeat dead threshold: 31  Network idle timeout: 30000  Network keepalive delay: 2000  Network reconnect delay: 2000  Heartbeat mode: LocalChecking O2CB heartbeat: Not activeDebug file system at /sys/kernel/debug: mounted另外：确认防火墙放行了对应的端口，本例是 tcp port 111115、配置 iscsi 存储[root@n149 ocfs2]# iscsiadm -m discovery -t st -p 10.50.200.41[root@n149 ocfs2]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea -p 192.168.131.101[root@n149 ocfs2]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea -p 192.168.131.102[root@n149 ocfs2]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea --login[root@n149 ocfs2]# yum install device-mapper-multipath -y[root@n149 ocfs2]# modprobe dm-multipath[root@n149 ocfs2]# mpathconf[root@n149 ocfs2]# cat /etc/multipath.confdefaults {    path_grouping_policy    multibus    path_checker            readsector0    rr_min_io_rq            10    max_fds                 8192    rr_weight               priorities    failback                immediate    no_path_retry           fail    user_friendly_names     yes}multipaths {    multipath {        wwid                    3600a098000a64d8b000002a4582305f8        alias                   test_block_a    }}[root@n149 ocfs2]# service multipathd start[root@n149 ocfs2]# chkconfig multipathd on[root@n149 ocfs2]# multipath -lltest_block_a (3600a098000a64d8b000002a4582305f8) dm-0 DELL,MD38xxisize=100G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active  |- 4:0:0:0 sda 8:0   active ready  running  `- 5:0:0:0 sdb 8:16  active ready  running  6、使用 iscsi 存储挂载参数请参考：man mkfs.ocfs2[root@n149 ~]# mkfs.ocfs2 -b 4K -C 64K -L testblocka -T datafiles -F /dev/mapper/test_block_amkfs.ocfs2 1.8.6Cluster stack: classic o2cbOverwriting existing ocfs2 partition.WARNING: Cluster check disabled.Proceed (y/N): yFilesystem Type of datafilesLabel: testblockaFeatures: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super xattr indexed-dirs refcount discontig-bgBlock size: 4096 (12 bits)Cluster size: 65536 (16 bits)Volume size: 107374182400 (1638400 clusters) (26214400 blocks)Cluster groups: 51 (tail covers 25600 clusters, rest cover 32256 clusters)Extent allocator size: 41943040 (10 groups)Journal size: 33554432Node slots: 8Creating bitmaps: doneInitializing superblock: doneWriting system files: doneWriting superblock: doneWriting backup superblock: 4 block(s)Formatting Journals: doneGrowing extent allocator: doneFormatting slot map: doneFormatting quota files: doneWriting lost+found: donemkfs.ocfs2 successful[root@n149 ~]# cat <<_EOF >>/etc/fstab/dev/mapper/test_block_a        /mnt/tba        ocfs2   defaults        0 0_EOF[root@n149 ~]# mkdir /mnt/tba[root@n149 ~]# mount -a[root@n149 ~]# df -h /mnt/tbaFilesystem                Size  Used Avail Use% Mounted on/dev/mapper/test_block_a  100G  586M  100G   1% /mnt/tba在另一个节点上执行：[root@n150 ~]# cat <<_EOF >>/etc/fstab/dev/mapper/test_block_a        /mnt/tba        ocfs2   defaults        0 0_EOF[root@n150 ~]# mkdir /mnt/tba[root@n150 ~]# mount -a[root@n150 ~]# df -h /mnt/tbaFilesystem                Size  Used Avail Use% Mounted on/dev/mapper/test_block_a  100G  586M  100G   1% /mnt/tba查看对应的日志：Nov 16 13:46:01 n149 kernel: o2hb: Heartbeat mode set to localNov 16 13:46:01 n149 o2hbmonitor: StartingNov 16 13:47:42 n149 kernel: OCFS2 1.8.0Nov 16 13:47:43 n149 kernel: o2dlm: Joining domain 55663D9EA9B0466D80626459E2B6AE9B ( 1 ) 1 nodesNov 16 13:47:43 n149 kernel: ocfs2: Mounting device (251,0) on (node 1, slot 0) with ordered data mode.Nov 16 13:48:16 n149 kernel: o2net: Connection to node n150.ocfs2.test (num 0) at 10.50.200.150:11111 shutdown, state 7Nov 16 13:48:18 n149 kernel: o2net: Connected to node n150.ocfs2.test (num 0) at 10.50.200.150:11111Nov 16 13:48:22 n149 kernel: o2dlm: Node 0 joins domain 55663D9EA9B0466D80626459E2B6AE9B ( 0 1 ) 2 nodesNov 16 13:45:43 n150 kernel: o2hb: Heartbeat mode set to localNov 16 13:45:43 n150 o2hbmonitor: StartingNov 16 13:48:18 n150 kernel: o2net: Accepted connection from node n149.ocfs2.test (num 1) at 10.50.200.149:11111Nov 16 13:48:22 n150 kernel: OCFS2 1.8.0Nov 16 13:48:22 n150 kernel: o2dlm: Joining domain 55663D9EA9B0466D80626459E2B6AE9B ( 0 1 ) 2 nodesNov 16 13:48:22 n150 kernel: ocfs2: Mounting device (251,0) on (node 0, slot 1) with ordered data mode.2个节点的 heartbeat 的状态也都变成了 Active[root@n149 ~]# service o2cb status Driver for "configfs": LoadedFilesystem "configfs": MountedStack glue driver: LoadedStack plugin "o2cb": LoadedDriver for "ocfs2_dlmfs": LoadedFilesystem "ocfs2_dlmfs": MountedChecking O2CB cluster "prdcluster": Online  Heartbeat dead threshold: 31  Network idle timeout: 30000  Network keepalive delay: 2000  Network reconnect delay: 2000  Heartbeat mode: LocalChecking O2CB heartbeat: ActiveDebug file system at /sys/kernel/debug: mounted7、测试文件读写【2个节点同时写一个文件】[root@n149 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done[root@n150 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done[root@n149 ~]# cat /mnt/tba/1.log   n149.ocfs2.test -> [Wed Nov 16 13:55:20 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:21 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:21 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:22 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:22 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:23 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:23 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:24 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:24 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:25 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:25 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:26 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:26 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:27 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:27 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:28 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:55:28 CST 2016]n150.ocfs2.test -> [Wed Nov 16 13:55:29 CST 2016]结论：符合预期。【一个节点写，另一个节点读】[root@n149 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done[root@n150 ~]# tail -f /mnt/tba/1.log n149.ocfs2.test -> [Wed Nov 16 13:57:42 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:43 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:44 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:45 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:46 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:47 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:48 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:49 CST 2016]n149.ocfs2.test -> [Wed Nov 16 13:57:50 CST 2016]^C结论：符合预期。三、问题解释1、cluster 处于 Offline 的状态[root@n150 ~]# mount -amount.ocfs2: Internal logic failure while trying to join the group[root@n150 ~]# service o2cb status       Checking O2CB cluster "prdcluster": Offline则请检查各节点的域名是否能正常解析，ocfs2 要用到 hostname 来提供服务。有一种情况是：上述节点的 hostname 为：n149.test.comn150.test.com而配置 ocfs2 是在 hosts 中写死了：10.50.200.149 n149.ocfs2.test10.50.200.150 n150.ocfs2.test而实际上，ocfs2 使用的却是 hostname，因而节点之间的连接异常。2、heartbeat 处于 Not active 状态挂载卷后 heartbeat 才会启动，因此，先操作后再观察。ZYXW、参考1、[ovirt-users] shared storage with iscsihttp://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi2、Linux : Oracle unbreakable kernel and OCFS2 on CentOS/RHEL6http://www.itechlounge.net/2014/08/linux-oracle-unbreakable-kernel-and-ocfs2-on-centosrhel6/3、Oracle Linux 6: Create an OCFS2 Cluster and Filesystemhttps://www.itfromallangles.com/2013/03/oracle-linux-6-create-an-ocfs2-cluster-and-filesystem/4、Installing And Configuring An OCFS2 Clustered File Systemhttp://www.shapeblue.com/installing-and-configuring-an-ocfs2-clustered-file-system/5、解决o2cb status时状态始终为offline(o2cb_ctl: Internal logic failure failed while adding RAC1)http://www.blogjava.net/KingKong/archive/2011/06/06/351826.html6、O2CB heartbeat: Not activehttps://oss.oracle.com/pipermail/ocfs2-users/2009-April/003477.html