如何搭建BeeGFS开发环境

这篇文章给大家介绍如何搭建BeeGFS开发环境，内容非常详细，感兴趣的小伙伴们可以参考借鉴，希望对大家能有所帮助。

安装软件包

我的测试环境使用CentOS 7， 安装BeeGFS的V6版本。

• 使用下载好的RPM包进行安装：

$ rpm -ivh beegfs_rpm/*.rpm$ rpm -qa | grep beebeegfs-meta-6.18-el7.x86_64beegfs-admon-6.18-el7.x86_64beegfs-utils-6.18-el7.x86_64beegfs-helperd-6.18-el7.x86_64beegfs-storage-6.18-el7.x86_64beegfs-opentk-lib-6.18-el7.x86_64beegfs-client-6.18-el7.noarchbeegfs-common-6.18-el7.noarchbeegfs-mgmtd-6.18-el7.x86_64

• 使用Yum安装：

$ wget https://www.beegfs.io/release/beegfs_6/dists/beegfs-rhel7.repo -O /etc/yum.repos.d/beegfs-rhel7.repo$ yum install -y beegfs-mgmtd beegfs-meta libbeegfs-ib beegfs-storage libbeegfs-ib beegfs-client beegfs-helperd beegfs-utils beegfs-admon$ yum install -y mdadm

初始化配置

初始化管理服务

sacd01 $ rm -rfv /beegfs_mgmtdsacd01 $ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /beegfs_mgmtdPreparing storage directory: /beegfs_mgmtd * Creating format.conf file...Updating config file: /etc/beegfs/beegfs-mgmtd.conf * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file...All done.

初始化元数据服务

• 两个元数据节点的ID分别选择2和3：

sacd01 $ rm /beegfs_meta/meta* -rfsacd01 $ /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs_meta/meta1 -s 2 -m 172.29.39.125Preparing storage directory: /beegfs_meta/meta1 * Creating format.conf file... * Creating server numeric ID file: /beegfs_meta/meta1/nodeNumIDUpdating config file: /etc/beegfs/beegfs-meta.conf * Setting management host: 172.29.39.125 * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... * Setting usage of extended attributes to: trueAll done.sacd02 $ rm /beegfs_meta/meta* -rfsacd02 $ /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs_meta/meta2 -s 3 -m 172.29.39.125Preparing storage directory: /beegfs_meta/meta2 * Creating format.conf file... * Creating server numeric ID file: /beegfs_meta/meta2/nodeNumIDUpdating config file: /etc/beegfs/beegfs-meta.conf * Setting management host: 172.29.39.125 * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... * Setting usage of extended attributes to: trueAll done.

初始化存储服务

sacd01 $ rm /beegfs_data/data* -rfsacd01 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data1 -s 4 -i 401 -m 172.29.39.125Preparing storage target directory: /beegfs_data/data1 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: /beegfs_data/data1/targetNumID * Creating server numeric ID file: /beegfs_data/data1/nodeNumIDUpdating config file: /etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.sacd01 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data2 -s 4 -i 402 -m 172.29.39.125Preparing storage target directory: /beegfs_data/data2 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: /beegfs_data/data2/targetNumID * Creating server numeric ID file: /beegfs_data/data2/nodeNumIDUpdating config file: /etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.sacd02 $ rm /beegfs_data/data* -rfsacd02 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data3 -s 5 -i 501 -m 172.29.39.125Preparing storage target directory: /beegfs_data/data3 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: /beegfs_data/data3/targetNumID * Creating server numeric ID file: /beegfs_data/data3/nodeNumIDUpdating config file: /etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.sacd02 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data4 -s 5 -i 502 -m 172.29.39.125Preparing storage target directory: /beegfs_data/data4 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: /beegfs_data/data4/targetNumID * Creating server numeric ID file: /beegfs_data/data4/nodeNumIDUpdating config file: /etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.

初始化客户端服务

sacd03 $ /opt/beegfs/sbin/beegfs-setup-client -m 172.29.39.125Updating config file: /etc/beegfs/beegfs-client.conf * Setting management host: 172.29.39.125All done.

配置服务网口

• 让让整个存储集群使用指定的网口进行通信：

sacd01 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.confsacd02 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.confsacd03 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf# Managementsacd01 $ vi /etc/beegfs/beegfs-mgmtd.confconnInterfacesFile                     = /etc/beegfs/network/beegfs-network-interface.conf# Metadatasacd01 $ vi /etc/beegfs/beegfs-meta.confconnInterfacesFile                     = /etc/beegfs/network/beegfs-network-interface.confsacd02 $ vi /etc/beegfs/beegfs-meta.confconnInterfacesFile                     = /etc/beegfs/network/beegfs-network-interface.conf# Storagesacd01 $ vi /etc/beegfs/beegfs-storage.confconnInterfacesFile                     = /etc/beegfs/network/beegfs-network-interface.confsacd02 $ vi vi /etc/beegfs/beegfs-storage.confconnInterfacesFile                     = /etc/beegfs/network/beegfs-network-interface.conf# Clientsacd03 $ vi /etc/beegfs/beegfs-client.confconnInterfacesFile                     = /etc/beegfs/network/beegfs-network-interface.conf

启动服务（客户端除外）

sacd01 $ systemctl start beegfs-mgmtdsacd01 $ systemctl start beegfs-metasacd02 $ systemctl start beegfs-metasacd01 $ systemctl start beegfs-storagesacd02 $ systemctl start beegfs-storagesacd03 $ beegfs-ctl --listtargets --nodetype=metadata --stateTargetID     Reachability  Consistency   NodeID========     ============  ===========   ======       2           Online         Good        2       3           Online         Good        3sacd03 $ beegfs-ctl --listtargets --nodetype=storage --stateTargetID     Reachability  Consistency   NodeID========     ============  ===========   ======     401           Online         Good        4     402           Online         Good        4     501           Online         Good        5     502           Online         Good        5

配置元数据服务镜像

sacd03 $ beegfs-ctl --addmirrorgroup --automatic --nodetype=metaNew mirror groups:BuddyGroupID   Node type Node============   ========= ====           1     primary        2 @ beegfs-meta sacd01 [ID: 2]               secondary        3 @ beegfs-meta sacd02 [ID: 3]Mirror buddy group successfully set: groupID 1 -> target IDs 2, 3sacd03 $ beegfs-ctl --listmirrorgroups --nodetype=meta     BuddyGroupID     PrimaryNodeID   SecondaryNodeID     ============     =============   ===============                1                 2                 3sacd03 $ beegfs-ctl --mirrormdsacd01 $ systemctl restart beegfs-metasacd02 $ systemctl restart beegfs-meta

配置和启动客户端服务

sacd03 $ vi /etc/beegfs/beegfs-client.conflogClientID                   = truelogLevel                      = 5#sysMountSanityCheckMS         = 11000sysMountSanityCheckMS          = 0sacd03 $ systemctl start beegfs-helperd && systemctl start beegfs-clientsacd03 $ systemctl status beegfs-helperd && systemctl status beegfs-clientsacd01 $ systemctl enable beegfs-mgmtd beegfs-meta beegfs-storagesacd02 $ systemctl enable beegfs-meta beegfs-storagesacd03 $ systemctl enable beegfs-client

查看服务使用的网口

$ beegfs-ctl --listnodes --nodetype=management --nicdetailssacd01 [ID: 1]   Ports: UDP: 8008; TCP: 8008   Interfaces:    + bond1.1039[ip addr: 172.29.39.125; type: TCP]Number of nodes: 1$ beegfs-ctl --listnodes --nodetype=metadata --nicdetailssacd01 [ID: 2]   Ports: UDP: 8005; TCP: 8005   Interfaces:    + bond1.1039[ip addr: 172.29.39.125; type: TCP]sacd02 [ID: 3]   Ports: UDP: 8005; TCP: 8005   Interfaces:    + bond1.1039[ip addr: 172.29.39.126; type: TCP]Number of nodes: 2Root: 2$ beegfs-ctl --listnodes --nodetype=storage --nicdetailssacd01 [ID: 4]   Ports: UDP: 8003; TCP: 8003   Interfaces:    + bond1.1039[ip addr: 172.29.39.125; type: TCP]sacd02 [ID: 5]   Ports: UDP: 8003; TCP: 8003   Interfaces:    + bond1.1039[ip addr: 172.29.39.126; type: TCP]Number of nodes: 2$ beegfs-ctl --listnodes --nodetype=client --nicdetailsC26A-5CD18762-sacd03 [ID: 5]   Ports: UDP: 8004; TCP: 0   Interfaces:    + bond1.1039[ip addr: 172.29.39.133; type: TCP]Number of nodes: 1

解决内核版本问题

启动客户端服务时，可能出现如下错误：

$ systemctl start beegfs-clientJob for beegfs-client.service failed because the control process exited with error code. See "systemctl status beegfs-client.service" and "journalctl -xe" for details.$ systemctl status beegfs-client.service -l● beegfs-client.service - Start BeeGFS Client   Loaded: loaded (/usr/lib/systemd/system/beegfs-client.service; enabled; vendor preset: disabled)   Active: failed (Result: exit-code) since Tue 2019-05-07 16:37:01 CST; 17s ago  Process: 134148 ExecStart=/etc/init.d/beegfs-client start (code=exited, status=1/FAILURE) Main PID: 134148 (code=exited, status=1/FAILURE)May 07 16:37:00 sacd03 systemd[1]: Starting Start BeeGFS Client...May 07 16:37:00 sacd03 beegfs-client[134148]: Starting BeeGFS Client:May 07 16:37:00 sacd03 beegfs-client[134148]: - Loading BeeGFS modulesMay 07 16:37:00 sacd03 beegfs-client[134148]: - BeeGFS module autobuildMay 07 16:37:01 sacd03 beegfs-client[134148]: Makefile:117: *** Linux kernel build directory not found. Please check if the kernel module development packages are installed for the current kernel version. (RHEL: kernel-devel; SLES: linux-kernel-headers, kernel-source; Debian: linux-headers).  Stop.May 07 16:37:01 sacd03 beegfs-client[134148]: make: *** [auto_rebuild] Error 2May 07 16:37:01 sacd03 systemd[1]: beegfs-client.service: main process exited, code=exited, status=1/FAILUREMay 07 16:37:01 sacd03 systemd[1]: Failed to start Start BeeGFS Client.May 07 16:37:01 sacd03 systemd[1]: Unit beegfs-client.service entered failed state.May 07 16:37:01 sacd03 systemd[1]: beegfs-client.service failed.

这是由于没有安装kernel-devel，没有内核头文件引起的。

或者是下面这个错误：

$ vi /var/log/beegfs-client.logMessage from syslogd@sacd03 at May  8 09:44:12 ... dracut:dracut: creation of /boot/initramfs-3.10.0-957.12.1.el7.x86_64.tmp failed(5) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] >> called. Path: /; EntryID: root(0) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: Messaging (RPC node)] >> Invalid mirror buddy group ID: 0(5) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] >> called. Path: n/a (no dentry); EntryID: n/a (no inode)(1) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] >> No storage targets known.(5) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] >> called. Path: /; EntryID: root(0) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: Messaging (RPC node)] >> Invalid mirror buddy group ID: 0(5) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] >> called. Path: n/a (no dentry); EntryID: n/a (no inode)(1) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] >> No storage targets known.$ ls /mnt/beegfs/ls: reading directory /mnt/beegfs/: Not a directory

这是BeeGFS 6.18在RHEL 3.10.0-862版本内核上的一个Bug引起的：

在所有节点上安装kernel-3.10.0-693版本的内核和开发包即可解决问题：

$ yum list kernelLoaded plugins: fastestmirror, langpacksLoading mirror speeds from cached hostfile * base: ap.stykers.moe * epel: mirrors.tuna.tsinghua.edu.cn * extras: ap.stykers.moe * updates: mirrors.tuna.tsinghua.edu.cnInstalled Packageskernel.x86_64                    3.10.0-957.5.1.el7                @updates Available Packageskernel.x86_64                    3.10.0-957.12.1.el7               updates  $ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-3.10.0-693.el7.x86_64.rpm$ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-devel-3.10.0-693.el7.x86_64.rpm# wget ftp://ftp.pbone.net/mirror/ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/kernel-3.10.0-693.el7.src.rpm$ rpm -ivh kernel* --force$ grep "menuentry " /boot/grub2/grub.cfg | awk -F"'" '{print $2}'CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)CentOS Linux (3.10.0-957.12.1.el7.x86_64) 7 (Core)CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core)$ vi /etc/default/grubGRUB_DEFAULT="CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)"$ grub2-mkconfig -o /boot/grub2/grub.cfg$ reboot$ rpm -evh kernel-3.10.0-957.12.1.el7.x86_64$ grep "menuentry " /boot/grub2/grub.cfg | awk -F"'" '{print $2}'CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core)

关于如何搭建BeeGFS开发环境就分享到这里了，希望以上内容可以对大家有一定的帮助，可以学到更多知识。如果觉得文章不错，可以把它分享出去让更多的人看到。