Difference between revisions of "Hydra"

From CSLabsWiki
(Install Packages)
 
(33 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
|ip_addr = 128.153.145.42
 
|ip_addr = 128.153.145.42
 
|contact_person = [[User:Jared|Jared Dunbar]]
 
|contact_person = [[User:Jared|Jared Dunbar]]
|last_update = ''Fall 2016''
+
|last_update = ''Spring 2017''
|services = Various
+
|services = VM Host
 
|category = Machines
 
|category = Machines
 
|handoff = no
 
|handoff = no
Line 10: Line 10:
 
{{Machine
 
{{Machine
 
| screenshot =
 
| screenshot =
| maintainer = [[User:Jared|Jared Dunbar]]
+
| maintainer = [[User:Jared|Jared Dunbar]] & [[User:bruskajp|James Bruska]]
 
| hostname = hydra.cslabs.clarkson.edu
 
| hostname = hydra.cslabs.clarkson.edu
| operating_system = Debian 8.2
+
| operating_system = Arch Linux
 
| interface1 = {{Network Interface | name=Clarkson Network | mac=0c:c4:7a:c6:e7:88 | ip=128.153.145.42}}
 
| interface1 = {{Network Interface | name=Clarkson Network | mac=0c:c4:7a:c6:e7:88 | ip=128.153.145.42}}
 
| cpuspecs = 2x 16 Core AMD Opteron 6376
 
| cpuspecs = 2x 16 Core AMD Opteron 6376
Line 19: Line 19:
 
}}
 
}}
   
Hydra is our newest VM host (as of September 2016
+
Hydra is our newest VM host (as of September 2016)
   
 
=Install=
 
=Install=
   
Do a basic debian install with the hard drives software raided in RAID 6, all of the device as /, then continue below
+
Do a basic Arch install with the hard drives software raided in RAID 6, all of the device as /
  +
  +
Make sure to install GRUB to ALL of the drives and NOT /dev/md0 - also protip: md0 is a partition, not strictly a block device, thus it does not need a mbr and you can mkfs.ext4 /dev/md0 without issue :D
   
 
==Install Packages==
 
==Install Packages==
   
  +
<pre style="background-color:#ffcccc">
<pre>
 
apt install htop iotop iftop lm-sensors vim sudo qemu-kvm libvirt-bin virtinst screen rsync tmux p7zip
+
pacman -Syu htop iotop iftop lm_sensors vim sudo qemu libvirtd screen rsync tmux p7zip
 
</pre>
 
</pre>
   
Change SSHd to use port 13699 instead of port 22. See
+
Change SSHd to use port 13699 in addition to port 22 (add a line below Port:22 with Port:13699). Edit
   
  +
<pre style="background-color:#ccccff">
<pre>
 
 
/etc/ssh/sshd_config
 
/etc/ssh/sshd_config
 
</pre>
 
</pre>
  +
  +
==User Configuration==
  +
  +
Add all of the users to the libvirtd and kvm groups that plan to use the VM host
   
 
==Network Configuration==
 
==Network Configuration==
   
We beed to create a network bridge for the VM's to connect on.
+
VM's require a network bridge in order to be able to communicate to the network.
   
<pre>
 
/etc/network/interfaces
 
</pre>
 
   
<pre>
 
source /etc/network/interfaces.d/*
 
   
  +
=Virtual Machine Core Allocation=
# The loopback network interface
 
auto lo
 
iface lo inet loopback
 
   
  +
These are cores that are allocated to VM's for efficiency.
# Auto Configure eth0 and eth1 (our two NIC's - the ethernet over the usb ports is actually a IPMI port)
 
auto eth0
 
auto eth1
 
   
  +
<pre style="background-color:#ffcccc">
# Bridge, set the IP here and bridge all the defined ports for communication
 
  +
virsh vcpupin <guestname> <local-core> <host core>
auto br0
 
iface br0 inet static
 
address 128.153.145.42
 
netmask 255.255.254.0
 
gateway 128.153.144.1
 
bridge_ports all
 
 
</pre>
 
</pre>
   
  +
Host cores are between 0 and 31 and local cores are from 0 to n-1 cores
=VM Information=
 
   
  +
You must run it for each core to each host core
Please note:
 
   
  +
{| class="wikitable"
Libvirt will automatically allocate RAM and CPU properly only to one CPU - but this can only happen if a VM is eating less than half of the RAM and half of the CPU power. VM's that have more than 16 cores or 32GB of ram may behave strangely or just not be as performant as you may expect. RAM latency will become an issue. VM's really should never use this much resources anyways, most being only a core or two, and somewhere between 256MB and 4GB of RAM.
 
  +
! CPU CORE
  +
! ALLOCATION
  +
|-
  +
|0
  +
|SYSTEM - DO NOT ALLOCATE
  +
|-
  +
|1
  +
|[[Keyserver]], [[scheduler]], [[honeypot]]
  +
|-
  +
|2
  +
|[[VRkour]], hunterGit
  +
|-
  +
|3
  +
|[[comm]], hwiki, [[Turist]]
  +
|-
  +
|4, 5
  +
|[[cacti]]
  +
|-
  +
|6, 7
  +
|[[gmx]]
  +
|-
  +
|24 to 31
  +
|[[colona-evolutionary-cpu]]
  +
|}
  +
  +
Free cores:
  +
<pre>
  +
6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23
  +
</pre>
   
 
=Hardware Information=
 
=Hardware Information=
Line 86: Line 107:
   
 
Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.
 
Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.
  +
  +
= Kernel Jankiness =
  +
  +
Under a Debian or Ubuntu kernel (even when custom compiled for the system by hand), shit goes haywire when you use GDB on this particular hardware configuration, for no reason. Thus, Arch Linux is the solution (for no apparent reason besides not having the symptoms)
  +
  +
Basically, the symptoms are that when debugging any process, we discovered that running si will cause the instruction pointer to rapidly become close to or become zero. We have no idea why this happens, but it's bad since it crashes the kernel. This includes if you are inside of a VM which is scary shit. Perhaps a CVE is in the works...

Latest revision as of 22:55, 10 September 2017

Hydra
IP Address(es): 128.153.145.42
Contact Person: Jared Dunbar
Last Update: Spring 2017
Services: VM Host


Hostname: hydra.cslabs.clarkson.edu
Operating system: Arch Linux
NIC 1: Clarkson Network
MAC: 0c:c4:7a:c6:e7:88
IP: 128.153.145.42
CPU: 2x 16 Core AMD Opteron 6376
RAM: 4x 16GB DDR3-1600 ECC


Hydra is our newest VM host (as of September 2016)

Install

Do a basic Arch install with the hard drives software raided in RAID 6, all of the device as /

Make sure to install GRUB to ALL of the drives and NOT /dev/md0 - also protip: md0 is a partition, not strictly a block device, thus it does not need a mbr and you can mkfs.ext4 /dev/md0 without issue :D

Install Packages

pacman -Syu htop iotop iftop lm_sensors vim sudo qemu libvirtd screen rsync tmux p7zip

Change SSHd to use port 13699 in addition to port 22 (add a line below Port:22 with Port:13699). Edit

/etc/ssh/sshd_config

User Configuration

Add all of the users to the libvirtd and kvm groups that plan to use the VM host

Network Configuration

VM's require a network bridge in order to be able to communicate to the network.


Virtual Machine Core Allocation

These are cores that are allocated to VM's for efficiency.

virsh vcpupin <guestname> <local-core> <host core>

Host cores are between 0 and 31 and local cores are from 0 to n-1 cores

You must run it for each core to each host core

CPU CORE ALLOCATION
0 SYSTEM - DO NOT ALLOCATE
1 Keyserver, scheduler, honeypot
2 VRkour, hunterGit
3 comm, hwiki, Turist
4, 5 cacti
6, 7 gmx
24 to 31 colona-evolutionary-cpu

Free cores:

6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23

Hardware Information

IMPORTANT - DO NOT BUMP OR JOSTLE CASE

The CPU0 heat sink cooler is not very well mounted. It is mounted properly, however if the server is bumped too hard, the cooler will pop off. Treat the server with extreme care ALWAYS

  • 2x 16 Core AMD Opeteron 6376
  • 4x 16GB DDR3-1600 ECC RAM
  • 5x 2TB Segate Constelation HDD in RAID 6
  • 2x Noctua CPU Coolers
  • Super Micro MBD-H8DG6-F-0 Motherboard EATX
  • Silver Stone 550 Watt 80+ Platinum Power Supply
  • Roswell 4u Server Chassis

Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.

Kernel Jankiness

Under a Debian or Ubuntu kernel (even when custom compiled for the system by hand), shit goes haywire when you use GDB on this particular hardware configuration, for no reason. Thus, Arch Linux is the solution (for no apparent reason besides not having the symptoms)

Basically, the symptoms are that when debugging any process, we discovered that running si will cause the instruction pointer to rapidly become close to or become zero. We have no idea why this happens, but it's bad since it crashes the kernel. This includes if you are inside of a VM which is scary shit. Perhaps a CVE is in the works...