Difference between revisions of "Hydra"

From CSLabsWiki
Jump to: navigation, search
(Virtual Machine Core Allocation)
Line 2: Line 2:
 
|ip_addr = 128.153.145.42
 
|ip_addr = 128.153.145.42
 
|contact_person = [[User:Jared|Jared Dunbar]] & [[User:bruskajp|James Bruska]]
 
|contact_person = [[User:Jared|Jared Dunbar]] & [[User:bruskajp|James Bruska]]
|last_update = ''Fall 2016''
+
|last_update = ''Spring 2017''
 
|services = VM Host
 
|services = VM Host
 
|category = Machines
 
|category = Machines
Line 12: Line 12:
 
| maintainer = [[User:Jared|Jared Dunbar]] & [[User:bruskajp|James Bruska]]
 
| maintainer = [[User:Jared|Jared Dunbar]] & [[User:bruskajp|James Bruska]]
 
| hostname = hydra.cslabs.clarkson.edu
 
| hostname = hydra.cslabs.clarkson.edu
| operating_system = Debian 8.2
+
| operating_system = Arch Linux
 
| interface1 = {{Network Interface | name=Clarkson Network | mac=0c:c4:7a:c6:e7:88 | ip=128.153.145.42}}
 
| interface1 = {{Network Interface | name=Clarkson Network | mac=0c:c4:7a:c6:e7:88 | ip=128.153.145.42}}
 
| cpuspecs = 2x 16 Core AMD Opteron 6376  
 
| cpuspecs = 2x 16 Core AMD Opteron 6376  
Line 23: Line 23:
 
=Install=
 
=Install=
  
Do a basic debian install with the hard drives software raided in RAID 6, all of the device as /, then continue below
+
Do a basic Arch install with the hard drives software raided in RAID 6, all of the device as /
 +
 
 +
Make sure to install GRUB to ALL of the drives and NOT /dev/md0 - also protip: md0 is a partition, not strictly a block device, thus it does not need a mbr and you can mkfs.ext4 /dev/md0 without issue :D
  
 
==Install Packages==
 
==Install Packages==
  
 
<pre style="background-color:#ffcccc">
 
<pre style="background-color:#ffcccc">
apt install htop iotop iftop lm-sensors vim sudo qemu-kvm libvirt-bin virtinst screen rsync tmux p7zip
+
pacman -Syu htop iotop iftop lm_sensors vim sudo qemu libvirtd screen rsync tmux p7zip
 
</pre>
 
</pre>
  
Change SSHd to use port 13699 instead of port 22. See
+
Change SSHd to use port 13699 in addition to port 22 (add a line below Port:22 with Port:13699). Edit
  
 
<pre style="background-color:#ccccff">
 
<pre style="background-color:#ccccff">
Line 37: Line 39:
 
</pre>
 
</pre>
  
==Network Configuration==
+
==User Configuration==
  
We beed to create a network bridge for the VM's to connect on.
+
Add all of the users to the libvirtd and kvm groups that plan to use the VM host
  
<pre style="background-color:#ccccff">
+
==Network Configuration==
/etc/network/interfaces
 
</pre>
 
 
 
<pre style="background-color:#ccffcc">
 
source /etc/network/interfaces.d/*
 
  
# The loopback network interface
+
VM's require a network bridge in order to be able to communicate to the network.
auto lo
 
iface lo inet loopback
 
  
# Auto Configure eth0 and eth1 (our two NIC's - the ethernet over the usb ports is actually a IPMI port)
 
auto eth0
 
auto eth1
 
  
# Bridge, set the IP here and bridge all the defined ports for communication
 
auto br0
 
iface br0 inet static
 
        address 128.153.145.42
 
        netmask 255.255.254.0
 
        gateway 128.153.144.1
 
        bridge_ports all
 
</pre>
 
  
 
=Virtual Machine Core Allocation=
 
=Virtual Machine Core Allocation=
Line 123: Line 107:
  
 
Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.
 
Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.
 +
 +
= Kernel Jankiness =
 +
 +
Under a Debian or Ubuntu kernel (even when custom compiled for the system by hand), shit goes haywire when you use GDB on this particular hardware configuration, for no reason. Thus, Arch Linux is the solution (for no apparent reason besides not having the symptoms)
 +
 +
Basically, the symptoms are that when debugging any process, we discovered that running si will cause the instruction pointer to rapidly become close to or become zero. We have no idea why this happens, but it's bad since it crashes the kernel. This includes if you are inside of a VM which is scary shit. Perhaps a CVE is in the works...

Revision as of 21:36, 31 January 2017

Hydra
IP Address(es): 128.153.145.42
Contact Person: Jared Dunbar & James Bruska
Last Update: Spring 2017
Services: VM Host


Hostname: hydra.cslabs.clarkson.edu
Operating system: Arch Linux
NIC 1: Clarkson Network
MAC: 0c:c4:7a:c6:e7:88
IP: 128.153.145.42
CPU: 2x 16 Core AMD Opteron 6376
RAM: 4x 16GB DDR3-1600 ECC


Hydra is our newest VM host (as of September 2016)

Install

Do a basic Arch install with the hard drives software raided in RAID 6, all of the device as /

Make sure to install GRUB to ALL of the drives and NOT /dev/md0 - also protip: md0 is a partition, not strictly a block device, thus it does not need a mbr and you can mkfs.ext4 /dev/md0 without issue :D

Install Packages

pacman -Syu htop iotop iftop lm_sensors vim sudo qemu libvirtd screen rsync tmux p7zip

Change SSHd to use port 13699 in addition to port 22 (add a line below Port:22 with Port:13699). Edit

/etc/ssh/sshd_config

User Configuration

Add all of the users to the libvirtd and kvm groups that plan to use the VM host

Network Configuration

VM's require a network bridge in order to be able to communicate to the network.


Virtual Machine Core Allocation

These are cores that are allocated to VM's for efficiency.

virsh vcpupin <guestname> <local-core> <host core>

Host cores are between 0 and 31 and local cores are from 0 to n-1 cores

You must run it for each core to each host core

CPU CORE ALLOCATION
0 SYSTEM - DO NOT ALLOCATE
1 Keyserver, scheduler, honeypot
2 VRkour, hunterGit
3 comm, hwiki, Turist
4, 5 cacti
6, 7 gmx
24 to 31 colona-evolutionary-cpu

Free cores:

6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23

Hardware Information

IMPORTANT - DO NOT BUMP OR JOSTLE CASE

The CPU0 heat sink cooler is not very well mounted. It is mounted properly, however if the server is bumped too hard, the cooler will pop off. Treat the server with extreme care ALWAYS

  • 2x 16 Core AMD Opeteron 6376
  • 4x 16GB DDR3-1600 ECC RAM
  • 5x 2TB Segate Constelation HDD in RAID 6
  • 2x Noctua CPU Coolers
  • Super Micro MBD-H8DG6-F-0 Motherboard EATX
  • Silver Stone 550 Watt 80+ Platinum Power Supply
  • Roswell 4u Server Chassis

Good to note is that there are 3 Ethernet ports. Two of them are NIC's (the pair right next to one another) and the one over the USB port is IPMI over Ethernet and has no connection to the kernel. Remote IPMI is disabled, so don't even bother with this.

Kernel Jankiness

Under a Debian or Ubuntu kernel (even when custom compiled for the system by hand), shit goes haywire when you use GDB on this particular hardware configuration, for no reason. Thus, Arch Linux is the solution (for no apparent reason besides not having the symptoms)

Basically, the symptoms are that when debugging any process, we discovered that running si will cause the instruction pointer to rapidly become close to or become zero. We have no idea why this happens, but it's bad since it crashes the kernel. This includes if you are inside of a VM which is scary shit. Perhaps a CVE is in the works...