OpenNebula

From GigaTux Wiki
Jump to navigation Jump to search

Debian base setup

  1. Install Debian
  2. Log in via SSH and su to root
  3. Change the path by using export PATH=$PATH:/usr/sbin
  4. Make sure GRUB is installed to all drives by using /usr/sbin/dpkg-reconfigure grub-pc
  5. Install general tools using apt-get install emacs-nox molly-guard elinks screen exim4-daemon-light lsof git iptables sudo rsyslog atop socat cryptsetup ntp unzip host openssh-server strace software-properties-common telnet reptyr bash-completion pv pigz pkg-config net-tools curl bc apt-transport-https ca-certificates apt-transport-https jq gnupg netcat cloud-test-utils arping
  6. Edit /etc/sudoers to have the line marc ALL=(ALL) NOPASSWD: ALL
  7. Change in /etc/ssh/sshd_config PermitRootLogin no
  8. groupadd --gid 2001 oneadmin
  9. useradd --uid 201 -g oneadmin -d /var/lib/one -m oneadmin
  10. Make datastore directories: mkdir /var/lib/one/datastores/ && mkdir /var/lib/one/datastores/100 && mkdir /var/lib/one/datastores/101 && chown -R oneadmin:oneadmin /var/lib/one/datastores
  11. lvcreate -n opennebula -L 4G vgnvme
  12. Add entries to fstab:
/dev/vgnvme/opennebula		/var/lib/one/datastores/100	    ext4	defaults,noatime,nodiratime,errors=remount-ro,noauto	0	2
/dev/vgsata/opennebula		/var/lib/one/datastores/101	    ext4	defaults,noatime,nodiratime,errors=remount-ro,noauto	0	2

cloudflared

  1. Get cloudflared from CloudFlare
  2. Run cloudflared tunnel create node0000X
  3. Add a /etc/cloudflared/config.yml file similar to:
tunnel: 540de4e5-0eb2-4c33-ad28-00dd25000000
credentials-file: /etc/cloudflared/540de4e5-0eb2-4c33-ad28-00dd25000000.json
logfile: /var/log/cloudflared.log
loglevel: info
proto-loglevel: error

ingress:
  - hostname: ssh-node0000X.gigatux.com
    service: ssh://localhost:22
  # Catch all
  - service: http_status:404
  1. Copy the relevant json configuratioon from the computer the tunnel was created on.
  2. Add a DNS entry in Cloudflare for ssh-node0000X.gigatux.com to go to the tunnel.

Node setup

  1. apt-get install bridge-utils dnsutils
  2. wget -q -O- http://downloads.opennebula.org/repo/Debian/repo.key | apt-key add -
  3. echo "deb http://downloads.opennebula.org/repo/6.0/Debian/10 stable opennebula" > /etc/apt/sources.list.d/opennebula.list
  4. apt-get install opennebula-node
  5. service libvirtd restart

The network interface used needs to be renamed to enp3s0 because this is how we've configured the Virtual Network in OpenNebula, so create a new file /etc/udev/rules.d/70-network.rules with contents similar to the below: 

 SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="d0:50:99:fc:ab:26", NAME="enp3s0"

Bootup script

  1. rc.local won't actually be run until we create a service, so create /etc/systemd/system/rc-local.service with the following contents:
#/etc/systemd/system/rc-local.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/rc.local

[Install]
WantedBy=multi-user.target

[Unit]
Wants=network-online.target
After=network-online.target
  1. systemctl enable rc-local

Put firewall commands in /etc/rc.local. Currently, these are: 

#!/bin/bash -x
THISHOSTNAME=node00005.gigatux.com

# Note, we use -w with iptables in case of any locks. We don't want it failing, and we're happy to wait for iptables to become available

# Flush rules
iptables -w -t mangle -F PREROUTING

######### RULES FOR us - node00005.gigatux.com #########

echo Mounting drives

mount /var/lib/one/datastores/100
mount /var/lib/one/datastores/101
chown oneadmin:oneadmin /var/lib/one/datastores/100
chown oneadmin:oneadmin /var/lib/one/datastores/101

echo Setting up firewall

iptables -w -F INPUT
iptables -w -P INPUT DROP

# ICMP
iptables -w -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
iptables -w -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
iptables -w -A INPUT -p icmp --icmp-type echo-request -m limit --limit 60/second -j ACCEPT
iptables -w -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
iptables -w -A INPUT -p icmp -j LOG --log-prefix ICMP:
iptables -w -A INPUT -p icmp -j DROP

# Whitelisted IPs
# Nebula front end
iptables -w -A INPUT -s 85.195.104.228 -j ACCEPT

# Allow all localhost
iptables -w -A INPUT -i lo -j ACCEPT

# Allow related, e.g. DNS
iptables -w -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Invalid flags
iptables -w -F badflags
iptables -w -X badflags
iptables -w -N badflags
iptables -w -A badflags -m limit --limit 15/minute -j LOG --log-prefix Badflags:
iptables -w -A badflags -j DROP

iptables -w -A INPUT -p tcp --tcp-flags ALL FIN,URG,PSH -j badflags
iptables -w -A INPUT -p tcp --tcp-flags ALL ALL -j badflags
iptables -w -A INPUT -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j badflags
iptables -w -A INPUT -p tcp --tcp-flags ALL NONE -j badflags
iptables -w -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j badflags
iptables -w -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j badflags

iptables -w -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -w -A INPUT -j LOG

echo Firewall setup complete

This may supersede the CSF instructions below.

CSF

Pretty sure CSF isn't running...

  1. On nodes: csf -a <front end address> "Nebula front-end"
  2. One front end: csf -a <node address> "node0000X"

Argo tunnel

Follow the instructions at the Bittylicious wiki.

SSH keys

We need to create a known_hosts file on the front end, so on that, run:

  1. su oneadmin
  2. ssh [email protected]
  3. log in and then exit
  4. Check that /var/lib/one/.ssh has a known_hosts file

Copy these to all nodes:

  1. scp -rp /var/lib/one/.ssh node00005.gigatux.com:/var/lib/one/

And verify ssh access in both directions from oneadmin account.

GRUB

  • Stop GRUB from probing all LVM devices by adding to /etc/default/grub: GRUB_DISABLE_OS_PROBER=true

Create the bridge

  1. brctl addbr br0
  2. brctl addif br0 enp3s0

Set up storage

    • TO DO **

Frontend setup

General config

Change /etc/one/sunstone-server.conf to have the :public_fireedge_endpoint to be the Cloudflare public URL

Customisation

Change /etc/sunstone-views.conf to have: 

 logo: https://gigatux.com/images/gigatux_logo_final_615x218_transparent.png

Change /etc/sunstone-views/mixed/cloud.yaml to have: 

< provision_logo: images/opennebula-5.0.png
---
> provision_logo: https://gigatux.com/images/gigatux_logo_final_615x218_transparent.png
26c27
<     cloud_vm_create: true
---
>     cloud_vm_create: false
38c39
<     show_attach_disk_advanced: true
---
>     show_attach_disk_advanced: false
40c41
<     show_attach_nic_advanced: true
---
>     show_attach_nic_advanced: false
64,65c65,66
<             flows: true
<             templates: true
---
>             flows: false
>             templates: false
85,87c86,88
<             VM.disk_resize: true
<             VM.attachdisk: true
<             VM.detachdisk: true
---
>             VM.disk_resize: false
>             VM.attachdisk: false
>             VM.detachdisk: false
89,90c90,91
<             VM.attachnic: true
<             VM.detachnic: true
---
>             VM.attachnic: false
>             VM.detachnic: false
113c114
<             quotas: true
---
>             quotas: false

Change /usr/lib/one/sunstone/public/css/custom.css to add: 

.support_not_connected {
   display: none !important;
}

#zonelector {
   display: none;
}

EVERYTHING BELOW HERE IS OLD

Pre-Installation Tasks

  1. elinks http://downloads.opennebula.org/
  2. apt-get install mysql-server git ruby-dev
  3. As MySQL root: CREATE USER 'oneadmin'@'localhost' IDENTIFIED BY 'PASSWORD' and GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin'@'localhost' IDENTIFIED BY 'password'

Install

  1. dpkg --install *.deb
  2. apt-get -f install (to get dependencies)
  3. /usr/share/opennebula/install_gems
  4. gem install nokogiri
  5. cd /usr/share/opennebula
  6. ./install_novnc.sh

Configure

Host configuration

  1. Copy /home/marc/git/xen/dist/install/usr/local/lib/xen/boot/pv-grub* on Flibble to /usr/lib/xen-4.1/boot/
  2. Ensure that /etc/xen/xend-config.sxp has the vnc-listen directive set to 0.0.0.0
  3. Also ensure that /etc/xen/xend-config.sxp has the following configured:
 (xend-relocation-server yes)
 (xen-api-server ((9363 'none') (unix)))
 (xend-tcp-xmlrpc-server yes)
 (xend-tcp-xmlrpc-server-address )
  1. Add opennebula to /etc/hosts
  2. apt-get install ruby sshfs python-yaml
  3. Copy key to /root/.ssh/authorized_keys
  4. groupadd and useradd commands as above
  5. usermod -a -G disk oneadmin
  6. mkdir /etc/xen/auto
  7. mkdir /etc/xen/vm
  8. ssh-keygen for oneadmin and add to authorized_keys on opennebula controller
  9. Log in to oneadmin@opennebula from all hosts to store the keys appropriately
  10. Add to /etc/sudoers:
 oneadmin  ALL=(ALL) NOPASSWD: /usr/sbin/xm *
 oneadmin  ALL=(ALL) NOPASSWD: /usr/sbin/xentop *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvs *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvcreate *
 oneadmin  ALL=(ALL) NOPASSWD: /sbin/lvremove *
 oneadmin  ALL=(ALL) NOPASSWD: /bin/dd *
  1. Mount the datastore by adding to /etc/fstab
 sshfs#oneadmin@opennebula:/var/lib/one/datastores/1   /var/lib/one/datastores/1   fuse defaults,allow_other,reconnect   0 0

Front end

  1. mkdir ~/.one
  2. echo "oneadmin:password" > ~/.one/one_auth
  3. chmod 600 ~/.one/one_auth
  4. Change :host: to 0.0.0.0 and :port: to 80 in /etc/one/sunstone-server.conf
  5. echo "Host *" >> ~/.ssh/config
  6. echo " StrictHostKeyChecking no" >> ~/.ssh/config
  7. Authorise self to SSH: cat /var/lib/one/.ssh/id_rsa.pub >> /var/lib/one/.ssh/authorized_keys

CXM

  1. cd ~
  2. git clone git://github.com/nagius/cxm
  3. cd git/cxm/misc
  4. cp -r opennebula/tm opennebula/vmm opennebula/cxmrc /var/lib/one/remotes
  5. cp cxm.conf /etc/
  6. Change XPATH directive in cxm.conf to read "xpath -e" <--- not there any more?
  7. Edit one-migrate in cxm's directory to have ONE_HOST set with the hostname of the frontend.
  8. scp one-migrate to all hosts in /usr/local/sbin/ and then run chmod u+x /usr/local/sbin/one-migrate on the hosts
  9. Change /etc/xen/cxm.conf to have:
POST_MIGRATION_HOOK="/usr/local/sbin/one-migrate"
  1. Change /var/lib/one/remotes/cxmrc to have:
DEBUG="True"
VG_NAME="lvm_r0"
LV_PREFIX="one"
VM_FQDN_TEMPLATE="%U-%V.one"
XPATH="xpath -e"

Root OpenNebula Configuration

  1. Make OpenNebula use the MySQL backend by editing /etc/one/oned.conf:
 DB = [ backend = "mysql",
   server  = "localhost",
   port    = 0,
   user    = "oneadmin",
   passwd  = "password",
   db_name = "opennebula" ]
  1. Go to /var/lib/one/remotes/vmm/xen4/xenrc and uncomment the line:
 export FORCE_DESTROY=yes
  1. Change /etc/one/oned.conf to uncomment the following (commenting the KVM lines):
 IM_MAD = [
     name       = "im_xen",
     executable = "one_im_ssh",
     arguments  = "xen" ]
 
 VM_MAD = [
     name       = "vmm_xen",
     executable = "one_vmm_exec",
     arguments  = "-t 15 -r 0 xen4",
     default    = "vmm_exec/vmm_exec_xen4.conf",
     type       = "xen" ]
  1. Also add cxm to the TM_MAD directive so it reads:
 TM_MAD = [
 executable = "one_tm",
 arguments  = "-t 15 -d dummy,lvm,shared,qcow2,ssh,vmware,iscsi,cxm" ]
  1. Add the cxm driver:
 VM_MAD = [
 name       = "vmm_cxm",
 executable = "one_vmm_exec",
 arguments  = "-t 15 -r 0 cxm",
 default    = "vmm_exec/vmm_exec_xen4.conf",
 type       = "xen" ]
  1. Change the default device prefix
 DEFAULT_DEVICE_PREFIX = "xvd"
  1. Change the default disk driver for phy: in /etc/one/vmm_exec/vmm_exec_xen4.conf :
 DISK = [ driver = "phy:" ]
  1. Add the following to /etc/rc.local:
 sunstone-server start

Networks

Create eth0.net with the following contents 

NAME = "Network Name"
TYPE = ranged

BRIDGE = eth0
NETWORK_SIZE = 30
NETWORK_ADDRESS = 192.168.1.224

GATEWAY = 192.168.1.254
DNS = 8.8.8.8

Run onevnet create eth0.net

Hosts

  • onehost create cloud1 --im xen --vm vmm_cxm --net dummy
  • onehost create cloud2 --im xen --vm vmm_cxm --net dummy
  • onehost enable cloud1
  • onehost enable cloud2

Datastores

  • Change the default datastore to use cxm by typing onedatastore update 1
 TM_MAD="cxm"
  • onedatastore create
 NAME="london1cxm"
 DS_MAD="fs"
 TM_MAD="cxm"
 NO_DECOMPRESS="yes"

Clusters

  • onecluster create London1
  • onecluster addhost London1 cloud1
  • onecluster addhost London1 cloud2
  • onecluster update 102
 SYSTEM_DS="X" where X is the cxm datastore
  • onecluster adddatastore London1 london1cxm

Images

  • Create debian.one in /images with the following
 NAME = "Deb 6 x86 Std"
 PATH = "/images/debian-opennebula.tar.gz"
 TYPE = OS
 DESCRIPTION = "Debian 6 (Squeeze) x86_64 with standard system utilities"
  • oneimage create debian.one -d <cxm datastore number>
  • Download the OpenNebula Virtual Router
  • gunzip it in /images
  • Create router.one in /images with the following
 NAME = "OpenNebula Router"
 PATH = "/images/router.img"
 TYPE = OS
 DESCRIPTION = "OpenNebula Router"
  • oneimage create router.one -d <cxm datastore number>

Templates

  • onetemplate create
 CPU="1"
 DISK=[
   FORMAT="ext3",
   IMAGE_ID="XX"
   SIZE="1024",
   TYPE="fs" ]
 DISK=[
   SIZE="128",
   TYPE="swap" ]
 GRAPHICS=[
   TYPE="vnc" ]
 MEMORY="128"
 NAME="Template Name"
 NIC=[
   NETWORK="Network Name" ]
 OS=[
   BOOTLOADER="/usr/lib/xen-4.1/bin/pygrub" ]
 RAW=[
   TYPE="xen" ]
 TEMPLATE_ID="X"
 VCPU="1"
  • onetemplate create (for the router)
 CONTEXT=[
   DHCP="YES",
   NTP="NO",
   PRIVNET="$NETWORK[TEMPLATE, NETWORK=\"Network Name\"]",
   TARGET="hdb",
   TEMPLATE="$TEMPLATE" ]
 CPU="1"
 DISK=[
   FORMAT="ext3",
   IMAGE_ID="YY"
   SIZE="2048",
   TYPE="raw" ]
 GRAPHICS=[
   TYPE="vnc" ]
 MEMORY="128"
 NAME="One Router"
 NIC=[
   NETWORK="Network Name" ]
 OS=[
   BOOTLOADER="/usr/lib/xen-4.1/bin/pygrub" ]
 RAW=[
   TYPE="xen" ]
 TEMPLATE_ID="X"
 VCPU="1"

Datastore Sharing

Use SSHFS to mount the datastore for the images on all hosts. Something like the following will do:

  1. mkdir /var/lib/one/datastores/<cxm datastore ID>
  2. sshfs -o allow_other -o reconnect oneadmin@opennebula:/var/lib/one/datastores/100 /var/lib/one/datastores/100

CXM post-configuration

  1. apt-get install python-twisted python-paramiko
  2. git clone https://github.com/nagius/cxm.git
  3. cp -rv cxm/lib/cxm /usr/lib/python2.7
  4. ln -s /usr/lib/xen-4.1/lib/python/xen /usr/lib/python2.7/

Router

To get the router up and running, a few changes need to be made within the VPS:

  1. loadkeys uk
  2. sed -i 's/KEYMAP="es"/KEYMAP="uk"/' /etc/rc.conf
  3. sed -i 's/Madrid/London/' /etc/rc.conf

Miscellaneous

  1. Download the vm_runn utility and save in ~oneadmin
Retrieved from "https://gigatux.com/wiki/index.php?title=OpenNebula&oldid=1081"