Difference between revisions of "Management"

From CSLabsWiki
(Added a bunch about running software.)
 
(129 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{services
 
{{services
  +
| screenshot = [[File:Cosi-management.png|200px]]
  +
| caption =
 
|ip_addr = 128.153.145.62
 
|ip_addr = 128.153.145.62
 
|contact_person = [[User:Jared|Jared Dunbar]]
 
|contact_person = [[User:Jared|Jared Dunbar]]
|last_update = ''January 2016''
+
|last_update = ''July 2017''
|services = server status indicator
+
|services = server status collection server
  +
|development_status = 30%
|category = VM
 
  +
|category = Machines
 
|handoff = no
 
|handoff = no
 
}}
 
}}
   
{{VM
 
| name = management
 
| screenshot =
 
| caption =
 
| maintainer = [[User:Jared|Jared Dunbar]]
 
| hostname = management
 
| operating_system = Debian 8.2
 
| ldap = none
 
| development_status = developing
 
| status = running
 
| vm_host = Felix
 
}}
 
   
'''Management''' is a VM on Felix that will be used for monitoring the status of VM's on other machines and the status of the hardware in the server room, ie. checking the CPU, RAM, and hard drive stats.
+
'''Management''' (stat3) is a vm used for monitoring the status of hosts in the server room, ie. checking the CPU, RAM, and hard drive stats, among other configurable things.
   
Each computer will have a startup executable that will send data periodically to '''Management''' which will be stored in a database and shown in an uptime page on a webpage that can easily be used to determine system uptime and service uptime.
+
Each computer in the server room that is configured sends data periodically which will be shown in an uptime page on a webpage that can easily be used to determine system uptime and service uptime among other things.
   
  +
Also, you can view COSI network stats at <del>http://management.cosi.clarkson.edu/cacti with the csguest user (and default password)</del> in raw data at http://stat.cosi.clarkson.edu/data since Cacti broke (again)
Currently installed on the VM are the following:
 
   
  +
=Installing Management Clients=
<pre>
 
htop openssh-client vim libmysql-java openjdk-7-jdk p7zip-full g++ sudo
 
</pre>
 
   
  +
Required Software: Git, g++, make
Required for the client side of the management software is:
 
   
  +
On Debian:
<pre>
 
  +
g++ top awk tail bash
 
  +
<pre style="background-color:#ffcccc">
  +
apt update && apt install make g++ git
 
</pre>
 
</pre>
 
The scripts rely on bash and the executable needs to be compiled for the architecture that it is made on.
 
   
  +
==Clone with Git==
The source code for the executable is available online at <https://github.com/jrddunbr/management-client>
 
   
  +
First, set git to allow all certificates, and get the files using Git.
The bash scripts are made wherever necessary but here are some functional examples:
 
   
  +
<pre style="background-color:#ffcccc">
CPU:
 
  +
git config --global http.sslVerify false
<pre>
 
  +
git clone https://gitlab.cosi.clarkson.edu/jared/manage2client.git
#!/bin/bash
 
DATA=$( top -bn 1 | awk '{print $9}' | tail -n +8 | awk '{s+=$1} END {print s}')
 
echo $DATA
 
/manage/management-client 128.153.145.62 80 cpu $DATA
 
 
</pre>
 
</pre>
   
  +
Re-secure the system by only accepting repos with certificates.
Used-Ram:
 
  +
<pre>
 
  +
<pre style="background-color:#ffcccc">
#!/bin/bash
 
  +
git config --global http.sslVerify true
FREE_DATA=`free -m | grep Mem`
 
DATA=`echo $FREE_DATA | cut -f3 -d' '`MB
 
echo $DATA
 
/manage/management-client 128.153.145.62 80 used-ram $DATA
 
 
</pre>
 
</pre>
   
  +
==Prepare files==
Total-Ram:
 
  +
<pre>
 
  +
Move the folder to the root.
#!/bin/bash
 
  +
FREE_DATA=`free -m | grep Mem`
 
  +
<pre style="background-color:#ffcccc">
DATA=`echo $FREE_DATA | cut -f2 -d' '`MB
 
  +
mv manage2client /manage
echo $DATA
 
/manage/management-client 128.153.145.62 80 total-ram $DATA
 
 
</pre>
 
</pre>
   
  +
Move the systemd service to the systemd serivces folder
These scripts expect management-client.cpp to be compiled as management-server and to be in the /manage folder (for simplicity, I tend to put them all in the same folder).
 
   
  +
<pre style="background-color:#ffcccc">
I also have one script that runs all of the scripts and this script is started by a Systemd Unit file located in /etc/systemd/system/manage.service:
 
  +
sudo mv /manage/manage.service /etc/systemd/system/manage.service
  +
</pre>
   
  +
==Configure system==
<pre>
 
[Unit]
 
Description=manage stuff
 
   
  +
If the hard drive you want to track is not /dev/sda1, select a different mount point to track in totaldisk.sh and useddisk.sh
[Service]
 
ExecStart=/bin/bash /manage/run.sh
 
   
  +
If you want to have virsh, edit run.sh, and un-comment the line with virsh.sh
[Install]
 
WantedBy=multi-user.target
 
   
  +
If you want to poll faster, change sleep from 30 to 5. Any faster, and the Linux scheduler will fall behind on busy boxes.
  +
  +
==Compile Management for your platform==
  +
  +
<pre style="background-color:#ffcccc">
  +
make
 
</pre>
 
</pre>
   
  +
==Enable Systemd Services==
It is easy to make more customized bash scripts that will complete other tasks. The compiled file has an expected input of ./management-client (IP) (PORT) (KEY) (VALUE) and this causes a key to go up, and saves at the value. When the server gets this as a rest call, the server reads it because it's in the 145 subnet and then sets it into the data structures of the program.
 
   
  +
<pre style="background-color:#ffcccc">
Unfortunately for the time being, the 145 subnet is a hard-coded thing. In future releases, as I have more time to finish this, it will become more functional and more features will arise.
 
  +
sudo systemctl enable manage
  +
sudo systemctl start manage
  +
</pre>
   
  +
==Whitelist==
The server side of the software is available at <https://github.com/jrddunbr/management-server> and is still a work in progress.
 
   
  +
Email dunbarj@clarkson.edu to get the server added to the whitelist
It requires the following to be installed:
 
   
  +
=Installing Management Server=
<pre>
 
  +
openjdk-7-jdk wget
 
  +
Start with an Arch VM
  +
  +
==Set Hostname==
  +
  +
Edit
  +
  +
<pre style="background-color:#ccccff">
  +
/etc/hostname
 
</pre>
 
</pre>
   
  +
Clear the contents and enter this on the first line, and save
You place the compiled .jar file in a handy place along with a few files:
 
   
  +
<pre style="background-color:#ccffcc">
<pre>
 
  +
management
index.html # a template HTML file that is used to list all of the servers, uptimes, and other data.
 
server.html # a template HTML file that is used to list one server and all of the associated key and value pairs that it has.
 
templates.yml # a template YAML file that is used to create all of the server specific YAML files. Once these are made, they will appear in the servers folder created in the root that the jar is run in,
 
master.yml # a file with future purposes to connect VM hosts with maintainers and to list which servers are essential to operation.
 
 
</pre>
 
</pre>
   
  +
==Set Network==
One downside to the whole system is that it depends on TALOS's HTTPS server to be running when this starts because it fetches the domain files. Future versions of the software might use a fallback mechanism where it copies the file to the hard drive as a backup.
 
   
  +
Copy example ethernet-static to netctl folder
Inside the servers folder, there are configurable per-server configs. The only active config is a disable: disable key that allows the server to be omitted from the server listing.\
 
   
  +
<pre style="background-color:#ffcccc">
Make sure that you check that your YAML files are parsed properly or I guarantee that the Java code will crash. There are a few good online checkers out there.
 
  +
cp /etc/netctl/examples/ethernet-static /etc/netctl/ehternet
  +
</pre>
   
  +
Edit
I made the startup script for the management server much the same as the client one, in fact I only changed the path to an executable SH file and changed the description slightly.
 
   
  +
<pre style="background-color:#ccccff">
The edited SH file that starts it is as follows:
 
  +
/etc/netctl/ethernet
  +
</pre>
   
  +
Clear the contents and set it to this:
<pre>
 
  +
cd /manage
 
  +
<pre style="background-color:#ccffcc">
java -jar management-server.jar
 
  +
Description='A basic static ethernet connection'
  +
Interface=ens3 # Make sure this is the interface or you won't have a network
  +
Connection=ethernet
  +
IP=static
  +
Address=('128.153.145.62/24')
  +
Gateway='128.153.145.1'
  +
DNS=('128.153.145.3')
 
</pre>
 
</pre>
   
  +
=Objectives=
As a helpful tip, here's how to start and stop Systemd unit files, do these:
 
  +
  +
*Create a monitoring system that can monitor all of the servers, battery backups, network, and also some temperature sensors placed through the server room at strategic locations
  +
*Notify computers of when to power down in a power outage
  +
*Create API's that can be used to interface the management platform
  +
  +
==Plans==
  +
  +
* Update ALL instances of Management to stat3 when client completed (and depricate the old versions - we still have versions of Management 1.0 and manage2client out there)
  +
* Create new server with authentication (both real encryption and perhaps OpenComputers compatible for Minecraft servers :P)
  +
* Configurable low power options
  +
* Email Notifications
  +
* Shutdowns
  +
* Sensors Interface - configurability is a must
  +
* Better web interface? Cookies? Logins? LDAP? PAM?
  +
* Ability for custom messages, custom dashboards?
   
  +
[[Category:Web Service]]
systemctl enable <name> for enabling units
 
systemctl disable <name> for disabling
 
systemctl start <name> for starting
 
systemctl stop <name> for stopping
 
systemctl status <name> (-l) for listing status, or -l for long form.
 

Latest revision as of 12:50, 29 July 2017

Management
Cosi-management.png
IP Address(es): 128.153.145.62
Contact Person: Jared Dunbar
Last Update: July 2017
Services: server status collection server


Management (stat3) is a vm used for monitoring the status of hosts in the server room, ie. checking the CPU, RAM, and hard drive stats, among other configurable things.

Each computer in the server room that is configured sends data periodically which will be shown in an uptime page on a webpage that can easily be used to determine system uptime and service uptime among other things.

Also, you can view COSI network stats at http://management.cosi.clarkson.edu/cacti with the csguest user (and default password) in raw data at http://stat.cosi.clarkson.edu/data since Cacti broke (again)

Installing Management Clients

Required Software: Git, g++, make

On Debian:

apt update && apt install make g++ git

Clone with Git

First, set git to allow all certificates, and get the files using Git.

git config --global http.sslVerify false
git clone https://gitlab.cosi.clarkson.edu/jared/manage2client.git

Re-secure the system by only accepting repos with certificates.

git config --global http.sslVerify true

Prepare files

Move the folder to the root.

mv manage2client /manage

Move the systemd service to the systemd serivces folder

sudo mv /manage/manage.service /etc/systemd/system/manage.service

Configure system

If the hard drive you want to track is not /dev/sda1, select a different mount point to track in totaldisk.sh and useddisk.sh

If you want to have virsh, edit run.sh, and un-comment the line with virsh.sh

If you want to poll faster, change sleep from 30 to 5. Any faster, and the Linux scheduler will fall behind on busy boxes.

Compile Management for your platform

make

Enable Systemd Services

sudo systemctl enable manage
sudo systemctl start manage

Whitelist

Email dunbarj@clarkson.edu to get the server added to the whitelist

Installing Management Server

Start with an Arch VM

Set Hostname

Edit

/etc/hostname

Clear the contents and enter this on the first line, and save

management

Set Network

Copy example ethernet-static to netctl folder

cp /etc/netctl/examples/ethernet-static /etc/netctl/ehternet

Edit

/etc/netctl/ethernet

Clear the contents and set it to this:

Description='A basic static ethernet connection'
Interface=ens3 # Make sure this is the interface or you won't have a network
Connection=ethernet
IP=static
Address=('128.153.145.62/24')
Gateway='128.153.145.1'
DNS=('128.153.145.3')

Objectives

  • Create a monitoring system that can monitor all of the servers, battery backups, network, and also some temperature sensors placed through the server room at strategic locations
  • Notify computers of when to power down in a power outage
  • Create API's that can be used to interface the management platform

Plans

  • Update ALL instances of Management to stat3 when client completed (and depricate the old versions - we still have versions of Management 1.0 and manage2client out there)
  • Create new server with authentication (both real encryption and perhaps OpenComputers compatible for Minecraft servers :P)
 * Configurable low power options
 * Email Notifications
 * Shutdowns
  • Sensors Interface - configurability is a must
  • Better web interface? Cookies? Logins? LDAP? PAM?
  • Ability for custom messages, custom dashboards?