OpenNebula

Debian base setup

1. Install Debian
2. Log in via SSH and su to root
3. Change the path by using export PATH=$PATH:/usr/sbin
4. Make sure GRUB is installed to all drives by using /usr/sbin/dpkg-reconfigure grub-pc
5. Install general tools using apt-get install emacs-nox molly-guard elinks screen exim4-daemon-light lsof git iptables sudo rsyslog atop socat cryptsetup ntp unzip host openssh-server strace software-properties-common telnet reptyr bash-completion pv pigz pkg-config net-tools curl bc apt-transport-https ca-certificates apt-transport-https jq gnupg netcat cloud-test-utils arping
6. Edit /etc/sudoers to have the line marc ALL=(ALL) NOPASSWD: ALL
7. Change in /etc/ssh/sshd_config PermitRootLogin no
8. groupadd --gid 2001 oneadmin
9. useradd --uid 201 -g oneadmin -d /var/lib/one -m oneadmin
10. Make datastore directories: mkdir /var/lib/one/datastores/ && mkdir /var/lib/one/datastores/100 && mkdir /var/lib/one/datastores/101 && chown -R oneadmin:oneadmin /var/lib/one/datastores
11. lvcreate -n opennebula -L 4G vgnvme
12. Disable mdadm monthly checks: dpkg-reconfigure mdadm
13. Add entries to fstab:

/dev/vgnvme/opennebula		/var/lib/one/datastores/100	    ext4	defaults,noatime,nodiratime,errors=remount-ro,noauto	0	2
/dev/vgsata/opennebula		/var/lib/one/datastores/101	    ext4	defaults,noatime,nodiratime,errors=remount-ro,noauto	0	2

cloudflared

1. Get cloudflared from CloudFlare
2. Run cloudflared tunnel create node0000X
3. Add a /etc/cloudflared/config.yml file similar to:

tunnel: 540de4e5-0eb2-4c33-ad28-00dd25000000
credentials-file: /etc/cloudflared/540de4e5-0eb2-4c33-ad28-00dd25000000.json
logfile: /var/log/cloudflared.log
loglevel: info
proto-loglevel: error

ingress:
  - hostname: ssh-node0000X.gigatux.com
    service: ssh://localhost:22
  # Catch all
  - service: http_status:404

1. Copy the relevant json configuratioon from the computer the tunnel was created on.
2. Add a DNS entry in Cloudflare for ssh-node0000X.gigatux.com to go to the tunnel.

Node setup

1. apt-get install bridge-utils dnsutils
2. wget -q -O- http://downloads.opennebula.org/repo/Debian/repo.key | apt-key add -
3. echo "deb http://downloads.opennebula.org/repo/6.0/Debian/10 stable opennebula" > /etc/apt/sources.list.d/opennebula.list
4. apt-get install opennebula-node
5. service libvirtd restart

The network interface used needs to be renamed to enp3s0 because this is how we've configured the Virtual Network in OpenNebula, so create a new file /etc/udev/rules.d/70-network.rules with contents similar to the below:

 SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="d0:50:99:fc:ab:26", NAME="enp3s0"

Bootup script

1. rc.local won't actually be run until we create a service, so create /etc/systemd/system/rc-local.service with the following contents:

#/etc/systemd/system/rc-local.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/rc.local

[Install]
WantedBy=multi-user.target

[Unit]
Wants=network-online.target
After=network-online.target

1. systemctl enable rc-local

Put firewall commands in /etc/rc.local. Currently, these are:

#!/bin/bash -x
THISHOSTNAME=node00005.gigatux.com

# Note, we use -w with iptables in case of any locks. We don't want it failing, and we're happy to wait for iptables to become available

# Flush rules
iptables -w -t mangle -F PREROUTING

######### RULES FOR us - node00005.gigatux.com #########

echo Mounting drives

mount /var/lib/one/datastores/100
mount /var/lib/one/datastores/101
chown oneadmin:oneadmin /var/lib/one/datastores/100
chown oneadmin:oneadmin /var/lib/one/datastores/101

echo Setting up firewall

iptables -w -F INPUT
iptables -w -P INPUT DROP

# ICMP
iptables -w -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
iptables -w -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
iptables -w -A INPUT -p icmp --icmp-type echo-request -m limit --limit 60/second -j ACCEPT
iptables -w -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
iptables -w -A INPUT -p icmp -j LOG --log-prefix ICMP:
iptables -w -A INPUT -p icmp -j DROP

# Whitelisted IPs
# Nebula front end
iptables -w -A INPUT -s 85.195.104.228 -j ACCEPT

# Allow all localhost
iptables -w -A INPUT -i lo -j ACCEPT

# Allow related, e.g. DNS
iptables -w -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Invalid flags
iptables -w -F badflags
iptables -w -X badflags
iptables -w -N badflags
iptables -w -A badflags -m limit --limit 15/minute -j LOG --log-prefix Badflags:
iptables -w -A badflags -j DROP

iptables -w -A INPUT -p tcp --tcp-flags ALL FIN,URG,PSH -j badflags
iptables -w -A INPUT -p tcp --tcp-flags ALL ALL -j badflags
iptables -w -A INPUT -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j badflags
iptables -w -A INPUT -p tcp --tcp-flags ALL NONE -j badflags
iptables -w -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j badflags
iptables -w -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j badflags

iptables -w -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -w -A INPUT -j LOG

echo Firewall setup complete

This may supersede the CSF instructions below.

CSF

Pretty sure CSF isn't running...

1. On nodes: csf -a <front end address> "Nebula front-end"
2. One front end: csf -a <node address> "node0000X"

Argo tunnel

Follow the instructions at the Bittylicious wiki.

SSH keys

We need to create a known_hosts file on the front end, so on that, run:

1. su oneadmin
2. ssh [email protected]
3. log in and then exit
4. Check that /var/lib/one/.ssh has a known_hosts file

Copy these to all nodes:

1. scp -rp /var/lib/one/.ssh node00005.gigatux.com:/var/lib/one/

And verify ssh access in both directions from oneadmin account.

GRUB

• Stop GRUB from probing all LVM devices by adding to /etc/default/grub: GRUB_DISABLE_OS_PROBER=true

Create the bridge

1. brctl addbr br0
2. brctl addif br0 enp3s0

Set up storage

• • TO DO **

Frontend setup

General config

Change /etc/one/sunstone-server.conf to have the :public_fireedge_endpoint to be the Cloudflare public URL

Customisation

Change /etc/sunstone-views.conf to have:

 logo: https://gigatux.com/images/gigatux_logo_final_615x218_transparent.png

Change /etc/sunstone-views/mixed/cloud.yaml to have:

< provision_logo: images/opennebula-5.0.png
---
> provision_logo: https://gigatux.com/images/gigatux_logo_final_615x218_transparent.png
26c27
<     cloud_vm_create: true
---
>     cloud_vm_create: false
38c39
<     show_attach_disk_advanced: true
---
>     show_attach_disk_advanced: false
40c41
<     show_attach_nic_advanced: true
---
>     show_attach_nic_advanced: false
64,65c65,66
<             flows: true
<             templates: true
---
>             flows: false
>             templates: false
85,87c86,88
<             VM.disk_resize: true
<             VM.attachdisk: true
<             VM.detachdisk: true
---
>             VM.disk_resize: false
>             VM.attachdisk: false
>             VM.detachdisk: false
89,90c90,91
<             VM.attachnic: true
<             VM.detachnic: true
---
>             VM.attachnic: false
>             VM.detachnic: false
113c114
<             quotas: true
---
>             quotas: false

Change /usr/lib/one/sunstone/public/css/custom.css to add:

.support_not_connected {
   display: none !important;
}

#zonelector {
   display: none;
}

Pre-Installation Tasks

1. elinks http://downloads.opennebula.org/
2. apt-get install mysql-server git ruby-dev
3. As MySQL root: CREATE USER 'oneadmin'@'localhost' IDENTIFIED BY 'PASSWORD' and GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin'@'localhost' IDENTIFIED BY 'password'

Install

1. dpkg --install *.deb
2. apt-get -f install (to get dependencies)
3. /usr/share/opennebula/install_gems
4. gem install nokogiri
5. cd /usr/share/opennebula
6. ./install_novnc.sh

Configure

Host configuration

1. Copy /home/marc/git/xen/dist/install/usr/local/lib/xen/boot/pv-grub* on Flibble to /usr/lib/xen-4.1/boot/
2. Ensure that /etc/xen/xend-config.sxp has the vnc-listen directive set to 0.0.0.0
3. Also ensure that /etc/xen/xend-config.sxp has the following configured:

 (xend-relocation-server yes)
 (xen-api-server ((9363 'none') (unix)))
 (xend-tcp-xmlrpc-server yes)
 (xend-tcp-xmlrpc-server-address )

1. Add opennebula to /etc/hosts
2. apt-get install ruby sshfs python-yaml
3. Copy key to /root/.ssh/authorized_keys
4. groupadd and useradd commands as above
5. usermod -a -G disk oneadmin
6. mkdir /etc/xen/auto
7. mkdir /etc/xen/vm
8. ssh-keygen for oneadmin and add to authorized_keys on opennebula controller
9. Log in to oneadmin@opennebula from all hosts to store the keys appropriately
10. Add to /etc/sudoers:

 oneadmin  ALL=(ALL) NOPASSWD: /usr/sbin/xm *
 oneadmin  ALL=(ALL) NOPASSWD: /usr/sbin/xentop *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvs *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvcreate *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvremove *
 oneadmin  ALL=(ALL) NOPASSWD: /bin/dd *

1. Mount the datastore by adding to /etc/fstab

 sshfs#oneadmin@opennebula:/var/lib/one/datastores/1   /var/lib/one/datastores/1   fuse defaults,allow_other,reconnect   0 0

Front end

1. mkdir ~/.one
2. echo "oneadmin:password" > ~/.one/one_auth
3. chmod 600 ~/.one/one_auth
4. Change :host: to 0.0.0.0 and :port: to 80 in /etc/one/sunstone-server.conf
5. echo "Host *" >> ~/.ssh/config
6. echo " StrictHostKeyChecking no" >> ~/.ssh/config
7. Authorise self to SSH: cat /var/lib/one/.ssh/id_rsa.pub >> /var/lib/one/.ssh/authorized_keys

CXM

1. cd ~
2. git clone git://github.com/nagius/cxm
3. cd git/cxm/misc
4. cp -r opennebula/tm opennebula/vmm opennebula/cxmrc /var/lib/one/remotes
5. cp cxm.conf /etc/
6. Change XPATH directive in cxm.conf to read "xpath -e" <--- not there any more?
7. Edit one-migrate in cxm's directory to have ONE_HOST set with the hostname of the frontend.
8. scp one-migrate to all hosts in /usr/local/sbin/ and then run chmod u+x /usr/local/sbin/one-migrate on the hosts
9. Change /etc/xen/cxm.conf to have:

POST_MIGRATION_HOOK="/usr/local/sbin/one-migrate"

1. Change /var/lib/one/remotes/cxmrc to have:

DEBUG="True"
VG_NAME="lvm_r0"
LV_PREFIX="one"
VM_FQDN_TEMPLATE="%U-%V.one"
XPATH="xpath -e"

Root OpenNebula Configuration

1. Make OpenNebula use the MySQL backend by editing /etc/one/oned.conf:

 DB = [ backend = "mysql",
   server  = "localhost",
   port    = 0,
   user    = "oneadmin",
   passwd  = "password",
   db_name = "opennebula" ]

1. Go to /var/lib/one/remotes/vmm/xen4/xenrc and uncomment the line:

 export FORCE_DESTROY=yes

1. Change /etc/one/oned.conf to uncomment the following (commenting the KVM lines):

 IM_MAD = [
     name       = "im_xen",
     executable = "one_im_ssh",
     arguments  = "xen" ]
 
 VM_MAD = [
     name       = "vmm_xen",
     executable = "one_vmm_exec",
     arguments  = "-t 15 -r 0 xen4",
     default    = "vmm_exec/vmm_exec_xen4.conf",
     type       = "xen" ]

1. Also add cxm to the TM_MAD directive so it reads:

 TM_MAD = [
 executable = "one_tm",
 arguments  = "-t 15 -d dummy,lvm,shared,qcow2,ssh,vmware,iscsi,cxm" ]

1. Add the cxm driver:

 VM_MAD = [
 name       = "vmm_cxm",
 executable = "one_vmm_exec",
 arguments  = "-t 15 -r 0 cxm",
 default    = "vmm_exec/vmm_exec_xen4.conf",
 type       = "xen" ]

1. Change the default device prefix

 DEFAULT_DEVICE_PREFIX = "xvd"

1. Change the default disk driver for phy: in /etc/one/vmm_exec/vmm_exec_xen4.conf :

 DISK = [ driver = "phy:" ]

1. Add the following to /etc/rc.local:

 sunstone-server start

Networks

Create eth0.net with the following contents

NAME = "Network Name"
TYPE = ranged

BRIDGE = eth0
NETWORK_SIZE = 30
NETWORK_ADDRESS = 192.168.1.224

GATEWAY = 192.168.1.254
DNS = 8.8.8.8

Run onevnet create eth0.net

Hosts

• onehost create cloud1 --im xen --vm vmm_cxm --net dummy
• onehost create cloud2 --im xen --vm vmm_cxm --net dummy
• onehost enable cloud1
• onehost enable cloud2

Datastores

• Change the default datastore to use cxm by typing onedatastore update 1

 TM_MAD="cxm"

• onedatastore create

 NAME="london1cxm"
 DS_MAD="fs"
 TM_MAD="cxm"
 NO_DECOMPRESS="yes"

Clusters

• onecluster create London1
• onecluster addhost London1 cloud1
• onecluster addhost London1 cloud2
• onecluster update 102

 SYSTEM_DS="X" where X is the cxm datastore

• onecluster adddatastore London1 london1cxm

Images

• Create debian.one in /images with the following

 NAME = "Deb 6 x86 Std"
 PATH = "/images/debian-opennebula.tar.gz"
 TYPE = OS
 DESCRIPTION = "Debian 6 (Squeeze) x86_64 with standard system utilities"

• oneimage create debian.one -d <cxm datastore number>
• Download the OpenNebula Virtual Router
• gunzip it in /images
• Create router.one in /images with the following

 NAME = "OpenNebula Router"
 PATH = "/images/router.img"
 TYPE = OS
 DESCRIPTION = "OpenNebula Router"

• oneimage create router.one -d <cxm datastore number>

Templates

• onetemplate create

 CPU="1"
 DISK=[
   FORMAT="ext3",
   IMAGE_ID="XX"
   SIZE="1024",
   TYPE="fs" ]
 DISK=[
   SIZE="128",
   TYPE="swap" ]
 GRAPHICS=[
   TYPE="vnc" ]
 MEMORY="128"
 NAME="Template Name"
 NIC=[
   NETWORK="Network Name" ]
 OS=[
   BOOTLOADER="/usr/lib/xen-4.1/bin/pygrub" ]
 RAW=[
   TYPE="xen" ]
 TEMPLATE_ID="X"
 VCPU="1"

• onetemplate create (for the router)

 CONTEXT=[
   DHCP="YES",
   NTP="NO",
   PRIVNET="$NETWORK[TEMPLATE, NETWORK=\"Network Name\"]",
   TARGET="hdb",
   TEMPLATE="$TEMPLATE" ]
 CPU="1"
 DISK=[
   FORMAT="ext3",
   IMAGE_ID="YY"
   SIZE="2048",
   TYPE="raw" ]
 GRAPHICS=[
   TYPE="vnc" ]
 MEMORY="128"
 NAME="One Router"
 NIC=[
   NETWORK="Network Name" ]
 OS=[
   BOOTLOADER="/usr/lib/xen-4.1/bin/pygrub" ]
 RAW=[
   TYPE="xen" ]
 TEMPLATE_ID="X"
 VCPU="1"

Datastore Sharing

Use SSHFS to mount the datastore for the images on all hosts. Something like the following will do:

1. mkdir /var/lib/one/datastores/<cxm datastore ID>
2. sshfs -o allow_other -o reconnect oneadmin@opennebula:/var/lib/one/datastores/100 /var/lib/one/datastores/100

CXM post-configuration

1. apt-get install python-twisted python-paramiko
2. git clone https://github.com/nagius/cxm.git
3. cp -rv cxm/lib/cxm /usr/lib/python2.7
4. ln -s /usr/lib/xen-4.1/lib/python/xen /usr/lib/python2.7/

Router

To get the router up and running, a few changes need to be made within the VPS:

1. loadkeys uk
2. sed -i 's/KEYMAP="es"/KEYMAP="uk"/' /etc/rc.conf
3. sed -i 's/Madrid/London/' /etc/rc.conf

Miscellaneous

1. Download the vm_runn utility and save in ~oneadmin

ProxySQL

Note - we're back to using SQLite because OpenNebula doesn't want to bootstrap a database

By default, OpenNebula uses sqlite for connectivity. We want to use our database cluster.

1. Download and install ProxySQL.

2. Edit the config to have:

mysql_servers =
(
	{
		address = "db-uk-cov-1.rwhr.bittylicious.chttps://forum.opennebula.io/t/error-when-bootstrapping-database-mysql/9942/3om" # no default, required . If port is 0 , address is interpred as a Unix Socket Domain
		port = 3306           # no default, required . If port is 0 , address is interpred as a Unix Socket Domain
		hostgroup = 0	        # no default, required
		status = "ONLINE"     # default: ONLINE
		weight = 100000            # default: 1
		compression = 0       # default: 0
		use_ssl = 1
		ssl_p2s_ca = "/etc/ssl/apache/cloudflare-origin.ecc.pem"
	},
	{
		address = "db-fr-rbx-1.rwhr.bittylicious.com" # no default, required . If port is 0 , address is interpred as a Unix Socket Domain
		port = 3306           # no default, required . If port is 0 , address is interpred as a Unix Socket Domain
		hostgroup = 0	        # no default, required
		status = "ONLINE"     # default: ONLINE
		weight = 1            # default: 1
		compression = 0       # default: 0
		use_ssl = 1
		ssl_p2s_ca = "/etc/ssl/apache/cloudflare-origin.ecc.pem"
	},
	{
		address = "db-fr-gra-1.rwhr.bittylicious.com" # no default, required . If port is 0 , address is interpred as a Unix Socket Domain
		port = 3306           # no default, required . If port is 0 , address is interpred as a Unix Socket Domain
		hostgroup = 0	        # no default, required
		status = "ONLINE"     # default: ONLINE
		weight = 1            # default: 1
		compression = 0       # default: 0
		use_ssl = 1
		ssl_p2s_ca = "/etc/ssl/apache/cloudflare-origin.ecc.pem"
	}

)

and

mysql_users:
(
	{
		username = "bittyuser" # no default , required
		password = "SH7Kx3SyQsmt" # default: ''
		default_hostgroup = 0 # default: 0
		active = 1            # default: 1
	},
)

And run the following commands in the local ProxySQL admin session:

UPDATE global_variables SET variable_value='monitor' WHERE variable_name='mysql-monitor_username';
UPDATE global_variables SET variable_value='MONITOR-PASSWORD' WHERE variable_name='mysql-monitor_password';

Make sure you add monitor users to the back end database, ensure firewalls allow the connection through etc.