Difference between revisions of "Hydra"

From CSLabsWiki
Jump to: navigation, search
(Virtual Machine Core Allocation)
 
(24 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
|ip_addr = 128.153.145.42
 
|ip_addr = 128.153.145.42
 
|contact_person = [[User:Jared|Jared Dunbar]]
 
|contact_person = [[User:Jared|Jared Dunbar]]
|last_update = ''Fall 2016''
+
|last_update = ''Spring 2017''
|services = Various
+
|services = VM Host
 
|category = Machines
 
|category = Machines
 
|handoff = no
 
|handoff = no
Line 10: Line 10:
 
{{Machine
 
{{Machine
 
| screenshot =  
 
| screenshot =  
| maintainer = [[User:Jared|Jared Dunbar]]
+
| maintainer = [[User:Jared|Jared Dunbar]] & [[User:bruskajp|James Bruska]]
 
| hostname = hydra.cslabs.clarkson.edu
 
| hostname = hydra.cslabs.clarkson.edu
| operating_system = Debian 8.2
+
| operating_system = Arch Linux
 
| interface1 = {{Network Interface | name=Clarkson Network | mac=0c:c4:7a:c6:e7:88 | ip=128.153.145.42}}
 
| interface1 = {{Network Interface | name=Clarkson Network | mac=0c:c4:7a:c6:e7:88 | ip=128.153.145.42}}
 
| cpuspecs = 2x 16 Core AMD Opteron 6376  
 
| cpuspecs = 2x 16 Core AMD Opteron 6376  
Line 19: Line 19:
 
}}
 
}}
  
Hydra is our newest VM host (as of September 2016
+
Hydra is our newest VM host (as of September 2016)
  
 
=Install=
 
=Install=
  
Do a basic debian install with the hard drives software raided in RAID 6, all of the device as /, then continue below
+
Do a basic Arch install with the hard drives software raided in RAID 6, all of the device as /
 +
 
 +
Make sure to install GRUB to ALL of the drives and NOT /dev/md0 - also protip: md0 is a partition, not strictly a block device, thus it does not need a mbr and you can mkfs.ext4 /dev/md0 without issue :D
  
 
==Install Packages==
 
==Install Packages==
  
<pre>
+
<pre style="background-color:#ffcccc">
apt install htop iotop iftop lm-sensors vim sudo qemu-kvm libvirt-bin virtinst screen rsync tmux p7zip
+
pacman -Syu htop iotop iftop lm_sensors vim sudo qemu libvirtd screen rsync tmux p7zip
 
</pre>
 
</pre>
  
Change SSHd to use port 13699 instead of port 22. See
+
Change SSHd to use port 13699 in addition to port 22 (add a line below Port:22 with Port:13699). Edit
  
<pre>
+
<pre style="background-color:#ccccff">
 
/etc/ssh/sshd_config
 
/etc/ssh/sshd_config
 
</pre>
 
</pre>
  
==Network Configuration==
+
==User Configuration==
  
We beed to create a network bridge for the VM's to connect on.
+
Add all of the users to the libvirtd and kvm groups that plan to use the VM host
  
<pre>
+
==Network Configuration==
/etc/network/interfaces
 
</pre>
 
 
 
<pre>
 
source /etc/network/interfaces.d/*
 
 
 
# The loopback network interface
 
auto lo
 
iface lo inet loopback
 
 
 
# Auto Configure eth0 and eth1 (our two NIC's - the ethernet over the usb ports is actually a IPMI port)
 
auto eth0
 
auto eth1
 
 
 
# Bridge, set the IP here and bridge all the defined ports for communication
 
auto br0
 
iface br0 inet static
 
        address 128.153.145.42
 
        netmask 255.255.254.0
 
        gateway 128.153.144.1
 
        bridge_ports all
 
</pre>
 
  
=VM Information=
+
VM's require a network bridge in order to be able to communicate to the network.
  
Please note:
 
  
Libvirt will automatically allocate RAM and CPU properly only to one CPU - but this can only happen if a VM is eating less than half of the RAM and half of the CPU power. VM's that have more than 16 cores or 32GB of ram may behave strangely or just not be as performant as you may expect. RAM latency will become an issue. VM's really should never use this much resources anyways, most being only a core or two, and somewhere between 256MB and 4GB of RAM.
 
  
 
=Virtual Machine Core Allocation=
 
=Virtual Machine Core Allocation=
  
These are cores that are allocated to VM's for efficiency. Do NOT allocate more than one task per core for efficiency sake!
+
These are cores that are allocated to VM's for efficiency.
  
<pre>
+
<pre style="background-color:#ffcccc">
 
virsh vcpupin <guestname> <local-core> <host core>
 
virsh vcpupin <guestname> <local-core> <host core>
 
</pre>
 
</pre>
Line 91: Line 69:
 
|-
 
|-
 
|1
 
|1
|[[Keyserver]]
+
|[[Keyserver]], [[scheduler]], [[honeypot]]
 
|-
 
|-
 
|2
 
|2
|[[VRkour]]
+
|[[VRkour]], hunterGit
 +
|-
 +
|3
 +
|[[comm]], hwiki, [[Turist]]
 
|-
 
|-
|20 to 23
+
|4, 5
|[[androbattery]]
+
|[[cacti]]
 +
|-
 +
|6, 7
 +
|[[gmx]]
 
|-
 
|-
 
|24 to 31
 
|24 to 31
|colona-evolutionary-cpu
+
|[[colona-evolutionary-cpu]]
 
|}
 
|}
 +
 +
Free cores:
 +
<pre>
 +
6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23
 +
</pre>
  
 
=Hardware Information=
 
=Hardware Information=
Line 118: Line 107:
  
 
Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.
 
Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.
 +
 +
= Kernel Jankiness =
 +
 +
Under a Debian or Ubuntu kernel (even when custom compiled for the system by hand), shit goes haywire when you use GDB on this particular hardware configuration, for no reason. Thus, Arch Linux is the solution (for no apparent reason besides not having the symptoms)
 +
 +
Basically, the symptoms are that when debugging any process, we discovered that running si will cause the instruction pointer to rapidly become close to or become zero. We have no idea why this happens, but it's bad since it crashes the kernel. This includes if you are inside of a VM which is scary shit. Perhaps a CVE is in the works...

Latest revision as of 21:55, 10 September 2017

Hydra
IP Address(es): 128.153.145.42
Contact Person: Jared Dunbar
Last Update: Spring 2017
Services: VM Host


Hostname: hydra.cslabs.clarkson.edu
Operating system: Arch Linux
NIC 1: Clarkson Network
MAC: 0c:c4:7a:c6:e7:88
IP: 128.153.145.42
CPU: 2x 16 Core AMD Opteron 6376
RAM: 4x 16GB DDR3-1600 ECC


Hydra is our newest VM host (as of September 2016)

Install

Do a basic Arch install with the hard drives software raided in RAID 6, all of the device as /

Make sure to install GRUB to ALL of the drives and NOT /dev/md0 - also protip: md0 is a partition, not strictly a block device, thus it does not need a mbr and you can mkfs.ext4 /dev/md0 without issue :D

Install Packages

pacman -Syu htop iotop iftop lm_sensors vim sudo qemu libvirtd screen rsync tmux p7zip

Change SSHd to use port 13699 in addition to port 22 (add a line below Port:22 with Port:13699). Edit

/etc/ssh/sshd_config

User Configuration

Add all of the users to the libvirtd and kvm groups that plan to use the VM host

Network Configuration

VM's require a network bridge in order to be able to communicate to the network.


Virtual Machine Core Allocation

These are cores that are allocated to VM's for efficiency.

virsh vcpupin <guestname> <local-core> <host core>

Host cores are between 0 and 31 and local cores are from 0 to n-1 cores

You must run it for each core to each host core

CPU CORE ALLOCATION
0 SYSTEM - DO NOT ALLOCATE
1 Keyserver, scheduler, honeypot
2 VRkour, hunterGit
3 comm, hwiki, Turist
4, 5 cacti
6, 7 gmx
24 to 31 colona-evolutionary-cpu

Free cores:

6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23

Hardware Information

IMPORTANT - DO NOT BUMP OR JOSTLE CASE

The CPU0 heat sink cooler is not very well mounted. It is mounted properly, however if the server is bumped too hard, the cooler will pop off. Treat the server with extreme care ALWAYS

  • 2x 16 Core AMD Opeteron 6376
  • 4x 16GB DDR3-1600 ECC RAM
  • 5x 2TB Segate Constelation HDD in RAID 6
  • 2x Noctua CPU Coolers
  • Super Micro MBD-H8DG6-F-0 Motherboard EATX
  • Silver Stone 550 Watt 80+ Platinum Power Supply
  • Roswell 4u Server Chassis

Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.

Kernel Jankiness

Under a Debian or Ubuntu kernel (even when custom compiled for the system by hand), shit goes haywire when you use GDB on this particular hardware configuration, for no reason. Thus, Arch Linux is the solution (for no apparent reason besides not having the symptoms)

Basically, the symptoms are that when debugging any process, we discovered that running si will cause the instruction pointer to rapidly become close to or become zero. We have no idea why this happens, but it's bad since it crashes the kernel. This includes if you are inside of a VM which is scary shit. Perhaps a CVE is in the works...