Data Center Dublin North - Open Compute Windmill deployment

Purpose

Deploy a small two through four node cluster of machines in a "hyper converged" infastructure method. The cluster must support virtual machines and should support containterized applications if possible.

Deploy an OpenStack "Compute Start Kit" four node group of machines. Primary services will be virtual machines images(KVM), container images (Podman or Docker).

OpenStack Docs

Technical Implementations

oVirt

Consider taking a look at oVirt

1. GlustFS config for storage.
2. Figure out oVirt.

Node disposition

Hostname	External	Internal	Service(s)
Node 1	192.168.0.201/24	192.168.100.201/24
Node 2	192.168.0.202/24	192.168.100.202/24
Node 3	192.168.0.203/24	192.168.100.203/24

StarlingX

The first attempts at running StarlingX failed. The hardware requirements of 64GB of RAM and two storage devices was not meet. I may revisit this configurtion by canabalizing RAM from one of the other nodes and attempting to use an NVMe drive as a PCIe device to meat the minimum requirements.

Kubernetes cluster

Disable swap comment out swap line in /etc/fstab Disable IPv6 https://linuxconfig.org/how-to-disable-ipv6-address-on-ubuntu-20-04-lts-focal-fossa update /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1" then run update-grub and reboot Network configuration bond0 mode: balance-alb

VMWare to NFS datastore using kernel NFS v4

The goal for this testing is to have an ESXi 6.7 host connected to a NFS server via NFS 4.1 protocol. The storage will be backed via a GlusterFS clustered volume.

Using virt-mgr I was able to get ESXi 6.7 installed as a guest machine.

This ran into quite a few issues with NFS and ESX. ESX does not support multipath NFS. ESX was completely unable to mount NFS volumes presented by NFSGanesha. ESX did connect to Kernel NFS but was "flakey".

Ubuntu Base OS configuration

1. Ubuntu 20.04
2. Configure IP addresses for service and backend networks.
3. apt install ssh
4. apt install firewalld
5. apt install smartmontools
6. apt install chrony
7. apt install xfsprogs - and configure XFS volumes on each drive.
8. sudo apt install glusterfs-server

9. apt install software-properties-common Then add the community GlusterFS PPA: add-apt-repository ppa:gluster/glusterfs-7 apt update Finally, install the packages: apt install glusterfs-server

Base CentOS Stream configuraiton

1. CentOS 8 minimal install
2. dnf install dnf-automatic
3. systemctl enable dnf-automatic-timer
4. edit /etc/dnf/automatic.conf change "apply_updates = yes"

MEI fix for Windmill 2.0 server

1. (modprobe -r mei echo "blacklist mei" >> /etc/modprobe.d/local-blacklist.conf echo "install mei /bin/false" >> /etc/modprobe.d/local-blacklist.conf)
2. echo "blacklist mei" >> /etc/modprobe.d/local-blacklist.conf
3. echo "install mei /bin/false" >> /etc/modprobe.d/local-blacklist.conf

Chrony NTP configuration

1. Create directory to use for NFS share

Network configuration

1. Use chronyd to synchronize to know time source
2. To be determined, firewall currently disabled. Recomendations are to have a dedicated network interface for gluster node communication and not firewall this interface on the network.

Drive configuration

1. mkfs.xfs /dev/vdb

GlusterFS configuration

1. dnf install centos-release-gluster
2. dnf install glusterfs-server
3. mkdir gv0
4. gluster volume create gv0 replica 2 192.168.0.101:/bricks/brick1/gv0/ 192.168.0.102:/bricks/brick1/gv0/
5. gluster volume start gv0

Kernel NFS configuration

1. dnf install nfs-utils
2. Edit /etc/exports /<directory name> 192.168.0.0/24(rw,all_squash)

NFS-Ganesha installation methodology

Having trouble with nfs-ganesha30 package on CentOS Stream. Trying nfs-ganesha28

1. dnf install dnf-automatic
2. edit /etc/dnf/automatic.conf change "apply_updates = yes"
3. dnf install centos-release-nfs-ganesha30
4. yum -y install nfs-ganesha nfs-ganesha-yum -y install nfs-ganesha nfs-ganesha-vfs
5. dnf --enablerepo=centos-nfs-ganesha3
6. sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-NFS-Ganesha-3.repo
7. dnf -y install centos-release-nfs-ganesha30

Argh... CentOS stream's package for nfs-ganesha30 does not include vfs. Switched to Ubuntu 20.04, please note much newer kernel

1. Install Ubuntu server 20.04
2. sudo apt install nfs-ganesha nfs-ganesha-vfs
3. Configure /etc/ganesha/ganesha.conf

EXPORT { Export_Id = 5; Path = /nfs; Pseudo = /nfs; Protocols = 4; Access_Type = RW; #Squash = root_squash; Sectype = none,sys; FSAL { Name = vfs; } }

Hardware deployment

Description	Quantity	Price	Ext. Price
240v outlet	2
Cover plate	2
Double gang box	2
Single gang box	2
Windmill R2.0 Server	4
Winbond 25q128bvfg	0
Intel 2680 CPU	2
Intel 2665 CPU	2
Micron 8GB DDR3	16
OEM (Dell) AMD 6450 GPU	1
Crucial/Micron Client SSDs CT1000MX500SSD1	4

Network configuration via ONOS.

1. Temperature monitoring via ipmi?
2. BMC does not provide console access.
3. DCIM, can I get it working?
4. Reboot issued through OS does not restart machine without a power cycle. Is it possible to write CoreBoot to the firmware instead?

Problem list

• Server reboot does not power back up. Appears to me a management engine bug in firmware. - Resolved by disabling mei kernel module. (modprobe -r mei echo "blacklist mei" >> /etc/modprobe.d/local-blacklist.conf echo "install mei /bin/false" >> /etc/modprobe.d/local-blacklist.conf)
• Wake on LAN not working in any way.

2024 Config

Packages

• Debian 12
• vim
• rasdaemon
• smartmontools

Network Configuration

Systemd-Networkd used for networking

systemctl enable systemd-networkd systemctl disable networking Create configuration files in /etc/systemd/network