OpenNebula

Node setup

1. Install Debian
2. apt-get install bridge-utils dnsutils
3. wget -q -O- http://downloads.opennebula.org/repo/Debian/repo.key | apt-key add -
4. echo "deb http://downloads.opennebula.org/repo/6.0/Debian/10 stable opennebula" > /etc/apt/sources.list.d/opennebula.list
5. apt-get install opennebula-node
6. service libvirtd restart

Bootup script

1. rc.local won't actually be run until we create a service, so create /etc/systemd/system/rc-local.service with the following contents:

#/etc/systemd/system/rc-local.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/rc.local

[Install]
WantedBy=multi-user.target

[Unit]
Wants=network-online.target
After=network-online.target

1. systemctl enable rc-local

Put firewall commands in /etc/rc.local

This may supersede the CSF instructions below.

CSF

1. On nodes: csf -a <front end address> "Nebula front-end"
2. One front end: csf -a <node address> "node0000X"

Argo tunnel

Follow the instructions at the Bittylicious wiki.

SSH keys

We need to create a known_hosts file on the front end, so on that, run:

1. su oneadmin
2. ssh [email protected]
3. log in and then exit
4. Check that /var/lib/one/.ssh has a known_hosts file

Copy these to all nodes:

1. scp -rp /var/lib/one/.ssh node00005.gigatux.com:/var/lib/one/

And verify ssh access in both directions from oneadmin account.

Create the bridge

1. brctl addif br0
2. brctl addif br0 enp3s0

Set up storage

• • TO DO **

Frontend setup

Change /etc/one/sunstone-server.conf to have the :public_fireedge_endpoint to be the Cloudflare public URL

Pre-Installation Tasks

1. groupadd --gid 2001 oneadmin
2. useradd --uid 201 -g oneadmin -d /var/lib/one -m oneadmin
3. elinks http://downloads.opennebula.org/
4. apt-get install mysql-server git ruby-dev
5. As MySQL root: CREATE USER 'oneadmin'@'localhost' IDENTIFIED BY 'PASSWORD' and GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin'@'localhost' IDENTIFIED BY 'password'

Install

1. dpkg --install *.deb
2. apt-get -f install (to get dependencies)
3. /usr/share/opennebula/install_gems
4. gem install nokogiri
5. cd /usr/share/opennebula
6. ./install_novnc.sh

Configure

Host configuration

1. Copy /home/marc/git/xen/dist/install/usr/local/lib/xen/boot/pv-grub* on Flibble to /usr/lib/xen-4.1/boot/
2. Ensure that /etc/xen/xend-config.sxp has the vnc-listen directive set to 0.0.0.0
3. Also ensure that /etc/xen/xend-config.sxp has the following configured:

 (xend-relocation-server yes)
 (xen-api-server ((9363 'none') (unix)))
 (xend-tcp-xmlrpc-server yes)
 (xend-tcp-xmlrpc-server-address )

1. Add opennebula to /etc/hosts
2. apt-get install ruby sshfs python-yaml
3. Copy key to /root/.ssh/authorized_keys
4. groupadd and useradd commands as above
5. usermod -a -G disk oneadmin
6. mkdir /etc/xen/auto
7. mkdir /etc/xen/vm
8. ssh-keygen for oneadmin and add to authorized_keys on opennebula controller
9. Log in to oneadmin@opennebula from all hosts to store the keys appropriately
10. Add to /etc/sudoers:

 oneadmin  ALL=(ALL) NOPASSWD: /usr/sbin/xm *
 oneadmin  ALL=(ALL) NOPASSWD: /usr/sbin/xentop *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvs *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvcreate *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvremove *
 oneadmin  ALL=(ALL) NOPASSWD: /bin/dd *

1. Mount the datastore by adding to /etc/fstab

 sshfs#oneadmin@opennebula:/var/lib/one/datastores/1   /var/lib/one/datastores/1   fuse defaults,allow_other,reconnect   0 0

Front end

1. mkdir ~/.one
2. echo "oneadmin:password" > ~/.one/one_auth
3. chmod 600 ~/.one/one_auth
4. Change :host: to 0.0.0.0 and :port: to 80 in /etc/one/sunstone-server.conf
5. echo "Host *" >> ~/.ssh/config
6. echo " StrictHostKeyChecking no" >> ~/.ssh/config
7. Authorise self to SSH: cat /var/lib/one/.ssh/id_rsa.pub >> /var/lib/one/.ssh/authorized_keys

CXM

1. cd ~
2. git clone git://github.com/nagius/cxm
3. cd git/cxm/misc
4. cp -r opennebula/tm opennebula/vmm opennebula/cxmrc /var/lib/one/remotes
5. cp cxm.conf /etc/
6. Change XPATH directive in cxm.conf to read "xpath -e" <--- not there any more?
7. Edit one-migrate in cxm's directory to have ONE_HOST set with the hostname of the frontend.
8. scp one-migrate to all hosts in /usr/local/sbin/ and then run chmod u+x /usr/local/sbin/one-migrate on the hosts
9. Change /etc/xen/cxm.conf to have:

POST_MIGRATION_HOOK="/usr/local/sbin/one-migrate"

1. Change /var/lib/one/remotes/cxmrc to have:

DEBUG="True"
VG_NAME="lvm_r0"
LV_PREFIX="one"
VM_FQDN_TEMPLATE="%U-%V.one"
XPATH="xpath -e"

Root OpenNebula Configuration

1. Make OpenNebula use the MySQL backend by editing /etc/one/oned.conf:

 DB = [ backend = "mysql",
   server  = "localhost",
   port    = 0,
   user    = "oneadmin",
   passwd  = "password",
   db_name = "opennebula" ]

1. Go to /var/lib/one/remotes/vmm/xen4/xenrc and uncomment the line:

 export FORCE_DESTROY=yes

1. Change /etc/one/oned.conf to uncomment the following (commenting the KVM lines):

 IM_MAD = [
     name       = "im_xen",
     executable = "one_im_ssh",
     arguments  = "xen" ]
 
 VM_MAD = [
     name       = "vmm_xen",
     executable = "one_vmm_exec",
     arguments  = "-t 15 -r 0 xen4",
     default    = "vmm_exec/vmm_exec_xen4.conf",
     type       = "xen" ]

1. Also add cxm to the TM_MAD directive so it reads:

 TM_MAD = [
 executable = "one_tm",
 arguments  = "-t 15 -d dummy,lvm,shared,qcow2,ssh,vmware,iscsi,cxm" ]

1. Add the cxm driver:

 VM_MAD = [
 name       = "vmm_cxm",
 executable = "one_vmm_exec",
 arguments  = "-t 15 -r 0 cxm",
 default    = "vmm_exec/vmm_exec_xen4.conf",
 type       = "xen" ]

1. Change the default device prefix

 DEFAULT_DEVICE_PREFIX = "xvd"

1. Change the default disk driver for phy: in /etc/one/vmm_exec/vmm_exec_xen4.conf :

 DISK = [ driver = "phy:" ]

1. Add the following to /etc/rc.local:

 sunstone-server start

Networks

Create eth0.net with the following contents

NAME = "Network Name"
TYPE = ranged

BRIDGE = eth0
NETWORK_SIZE = 30
NETWORK_ADDRESS = 192.168.1.224

GATEWAY = 192.168.1.254
DNS = 8.8.8.8

Run onevnet create eth0.net

Hosts

• onehost create cloud1 --im xen --vm vmm_cxm --net dummy
• onehost create cloud2 --im xen --vm vmm_cxm --net dummy
• onehost enable cloud1
• onehost enable cloud2

Datastores

• Change the default datastore to use cxm by typing onedatastore update 1

 TM_MAD="cxm"

• onedatastore create

 NAME="london1cxm"
 DS_MAD="fs"
 TM_MAD="cxm"
 NO_DECOMPRESS="yes"

Clusters

• onecluster create London1
• onecluster addhost London1 cloud1
• onecluster addhost London1 cloud2
• onecluster update 102

 SYSTEM_DS="X" where X is the cxm datastore

• onecluster adddatastore London1 london1cxm

Images

• Create debian.one in /images with the following

 NAME = "Deb 6 x86 Std"
 PATH = "/images/debian-opennebula.tar.gz"
 TYPE = OS
 DESCRIPTION = "Debian 6 (Squeeze) x86_64 with standard system utilities"

• oneimage create debian.one -d <cxm datastore number>
• Download the OpenNebula Virtual Router
• gunzip it in /images
• Create router.one in /images with the following

 NAME = "OpenNebula Router"
 PATH = "/images/router.img"
 TYPE = OS
 DESCRIPTION = "OpenNebula Router"

• oneimage create router.one -d <cxm datastore number>

Templates

• onetemplate create

 CPU="1"
 DISK=[
   FORMAT="ext3",
   IMAGE_ID="XX"
   SIZE="1024",
   TYPE="fs" ]
 DISK=[
   SIZE="128",
   TYPE="swap" ]
 GRAPHICS=[
   TYPE="vnc" ]
 MEMORY="128"
 NAME="Template Name"
 NIC=[
   NETWORK="Network Name" ]
 OS=[
   BOOTLOADER="/usr/lib/xen-4.1/bin/pygrub" ]
 RAW=[
   TYPE="xen" ]
 TEMPLATE_ID="X"
 VCPU="1"

• onetemplate create (for the router)

 CONTEXT=[
   DHCP="YES",
   NTP="NO",
   PRIVNET="$NETWORK[TEMPLATE, NETWORK=\"Network Name\"]",
   TARGET="hdb",
   TEMPLATE="$TEMPLATE" ]
 CPU="1"
 DISK=[
   FORMAT="ext3",
   IMAGE_ID="YY"
   SIZE="2048",
   TYPE="raw" ]
 GRAPHICS=[
   TYPE="vnc" ]
 MEMORY="128"
 NAME="One Router"
 NIC=[
   NETWORK="Network Name" ]
 OS=[
   BOOTLOADER="/usr/lib/xen-4.1/bin/pygrub" ]
 RAW=[
   TYPE="xen" ]
 TEMPLATE_ID="X"
 VCPU="1"

Datastore Sharing

Use SSHFS to mount the datastore for the images on all hosts. Something like the following will do:

1. mkdir /var/lib/one/datastores/<cxm datastore ID>
2. sshfs -o allow_other -o reconnect oneadmin@opennebula:/var/lib/one/datastores/100 /var/lib/one/datastores/100

CXM post-configuration

1. apt-get install python-twisted python-paramiko
2. git clone https://github.com/nagius/cxm.git
3. cp -rv cxm/lib/cxm /usr/lib/python2.7
4. ln -s /usr/lib/xen-4.1/lib/python/xen /usr/lib/python2.7/

Router

To get the router up and running, a few changes need to be made within the VPS:

1. loadkeys uk
2. sed -i 's/KEYMAP="es"/KEYMAP="uk"/' /etc/rc.conf
3. sed -i 's/Madrid/London/' /etc/rc.conf

Miscellaneous

1. Download the vm_runn utility and save in ~oneadmin