Difference between revisions of "Management"
(→Prepare files) |
(→Enable Systemd Services) |
||
Line 70: | Line 70: | ||
==Enable Systemd Services== | ==Enable Systemd Services== | ||
− | <pre> | + | <pre style="background-color:#ffcccc"> |
sudo systemctl enable manage | sudo systemctl enable manage | ||
sudo systemctl start manage | sudo systemctl start manage |
Revision as of 13:23, 2 September 2016
Management | |
![]() | |
IP Address(es): | 128.153.145.62 |
Contact Person: | Jared Dunbar |
Last Update: | February 2016 |
Services: | server status indicator |
Hostname: | management |
Operating system: | Armbian (Debian) Jessie (kernel 4.4.1-sunxi) |
NIC 1: | eth0 MAC: 02:8e:08:41:65:6a IP: 128.153.145.62 |
CPU: | Hard Float Dual Core Allwinner A20 armv7l, Mali 400 MP2 |
RAM: | 1GB DDR3 ECC |
Management (manage2server) is a vm used for monitoring the status of VM's on other machines and the status of the hardware in the server room, ie. checking the CPU, RAM, and hard drive stats, among other configurable things.
Each computer in the server room that is configured sends data periodically to Management which will be shown in an uptime page on a webpage that can easily be used to determine system uptime and service uptime among other things.
Also, you can view COSI network stats at http://management.cosi.clarkson.edu/cacti with the csguest user (and default password)
Contents
Installing Management Clients
Clone with Git
First, set git to allow all certificates, and get the files using Git.
git config --global http.sslVerify false git clone https://gitlab.cosi.clarkson.edu/jared/manage2client.git
Optionally, re-secure the system by only accepting repos with certificates.
git config --global http.sslVerify true
Prepare files
Move the folder to the root.
mv manage2client /manage
Move the systemd service to the systemd serivces folder
sudo mv /manage/manage.service /etc/systemd/system/manage.service
Configure system
If the hard drive you want to track is not /dev/sda1, select a different mount point to track in totaldisk.sh and useddisk.sh
If you want to have virsh, edit run.sh, and un-comment the line with virsh.sh
If you want to poll faster, change sleep from 30 to 5. Any faster, and the Linux scheduler will fall behind on busy boxes.
Enable Systemd Services
sudo systemctl enable manage sudo systemctl start manage
Sit back and relax! It should show up on a running manage2server momentarily.
Installing Management Server
Start with an Arch VM
Set Hostname
Edit
/etc/hostname
Clear the contents and enter this on the first line, and save
management
Set Network
Copy example ethernet-static to netctl folder
cp /etc/netctl/examples/ethernet-static /etc/netctl/ehternet
Edit
/etc/netctl/ethernet
Clear the contents and set it to this:
Description='A basic static ethernet connection' Interface=ens3 # Make sure this is the interface or you won't have a network Connection=ethernet IP=static Address=('128.153.145.62/24') Gateway='128.153.145.1' DNS=('128.153.145.3')
Old Management (1.0)
Management is a SBC (single board computer) used for monitoring the status of VM's on other machines and the status of the hardware in the server room, ie. checking the CPU, RAM, and hard drive stats, among other configurable things.
Each computer in the server room that is configured sends data periodically to Management which will be shown in an uptime page on a webpage that can easily be used to determine system uptime and service uptime among other things.
Also, you can view COSI network stats at http://management.cosi.clarkson.edu:8080/cacti with the csguest user (and default password)
Currently installed on the machine are the following:
htop openssh-client vim openjdk-7-jdk p7zip-full g++ sudo git upower apcupsd apache2 cacti mysql-server
If you need to update the maintainers list for a server, please contact Jared for the instructions on how to do so.
If there is an invalid key:value pair sent from your server and it is now in the listing, contact Jared if you want it removed.
If you want to add a server, install the management client, and then contact Jared to add the device to the list.
Client Side (runs on a server)
Requirements
g++ top awk tail bash sed
The source code for the client executable is available online at https://github.com/jrddunbr/management-client
The bash scripts are made wherever necessary (it's expandable and each server can theoretically have as many keys as it wants, each data parameter is stored as a key) and here are some functional examples:
CPU:
#!/bin/bash DATA=$( top -bn2 | \ grep "Cpu(s)" | \ sed -n '1!p' | \ sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \ awk '{print 100 - $1}') echo $DATA /manage/management-client 128.153.145.62 80 cpu $DATA
Used-Ram:
#!/bin/bash read _1 MEMTOTAL _2 <<< "$(head -n 1 /proc/meminfo)" read _1 MEMAVAIL _2 <<< "$(tail -n +3 /proc/meminfo | head -n 1)" DATA="$(( (MEMTOTAL - MEMAVAIL) / 1024 ))MB" echo $DATA /manage/management-client 128.153.145.62 80 used-ram $DATA
Total-Ram:
#!/bin/bash read _1 MEMTOTAL _2 <<< "$(head -n 1 /proc/meminfo)" DATA="$(( MEMTOTAL / 1024 ))MB" echo $DATA /manage/management-client 128.153.145.62 80 total-ram $DATA
Uptime:
#!/bin/bash DATA=$(uptime -p | sed -e 's/ /_/g') echo $DATA /manage/management-client 128.153.145.62 80 uptime "$DATA"
Virsh:
Only install this one if you have a VM server running. This MUST be run as the root user.
while read ID NAME STAT; do echo "NAME=$NAME, STAT=$STAT"; STAT=$(echo $STAT | sed -e 's/ /_/g') /manage/management-client 128.153.145.62 80 $NAME $STAT done <<< "$(virsh list --all | tail -n +3)"
Compiling Managemnet
These scripts expect management-client.cpp to be compiled as
g++ management-client.cpp -o management-client --std=c++11
and to be in the /manage folder (for simplicity, I tend to put them all in the same folder).
management.cpp
/* * File: management-client.cpp * Author: jared * * Created on January 21, 2016, 3:50 PM */ #include <cstdlib> #include <iostream> #include <fstream> #include <sys/socket.h> #include <arpa/inet.h> #include <netdb.h> using namespace std; class tcp_module { private: string address; int port; struct sockaddr_in server; int sock; public: tcp_module(string, int); void sendTcp(string); }; tcp_module::tcp_module(string addr, int portNumber) { address = addr; port = portNumber; sock = socket(AF_INET, SOCK_STREAM, 0); if (sock == -1) { cerr << "Error creating the socket\nExiting\n"; exit(1); } if (inet_addr(address.c_str()) == -1) { struct hostent *host; if ((host = gethostbyname(address.c_str()))) { cerr << "Error resolving host " << address << "\nPlease use an IP if you are not, exiting\n"; exit(1); } } server.sin_addr.s_addr = inet_addr(address.c_str()); server.sin_family = AF_INET; server.sin_port = htons(port); if (connect(sock, ((struct sockaddr*) &server), sizeof (server)) < 0) { cerr << "Error connecting to host. Exiting.\n"; exit(1); } } void tcp_module::sendTcp(string data) { if (send(sock, data.c_str(), data.length(), 0) < 0) { cerr << "Error sending data. Exiting\n"; exit(1); } } bool isConnected = false; void sendKey(tcp_module connection, string key, string data) { string send = "GET /" + key + "/" + data + " HTTP/1.1\r\n\r\n"; connection.sendTcp(send); } int main(int argc, char** argv) { if(argc != 5) { cerr << "Error, not enough arguments!\n"; cout << "\n Usage:\n\n./[executable] host port key value\n"; for(int i = 0; i < argc; i++) { cout << argv[i] << " "; } cout << "\n"; exit(1); } string key = argv[3]; string data = argv[4]; string host = argv[1]; int port = atoi(argv[2]); tcp_module tcp(host, port); sendKey(tcp, key, data); return 0; }
Startup
I also have one script that runs all of the client scripts. The Bash script that runs all the other bash scripts looks a lot like this:
Bash Start Script
/manage/run.sh
#!/bin/bash cd /manage while true do /manage/cpu.sh & /manage/used-ram.sh & /manage/total-ram.sh & /manage/uptime.sh & # /manage/virsh.sh & # this is for if you have a virsh virtual machine system running, to monitor VM stats. sleep 20 done
Extensibility
It is easy to make more customized bash scripts that will complete other tasks. The compiled file has an expected input of ./management-client (IP) (PORT) (KEY_NAME) (VALUE) and this causes a key to go up, and saves at the value. When the server gets this as a rest call, the server reads it because it's listed in an internal file.
One thing to note is that it takes only 5 arguments: the executable, the IP, port, key, and value. Each one has to have no spaces in it. If you want your key or value to have a space, place an underscore and it will be replaced with a space. The error it tells you is basic, but informative. You must use an IP for the ip field, it will not accept hostnames.
Technically, management can recieve key:value pairs from servers outside of the server room or even Clarkson if it is configured in the networks.txt file and also in the master.yml file. (to be changed later)
Server Side (management itself)
Requirements
The server side of the software is available at https://github.com/jrddunbr/management-server and is still a work in progress.
It requires the following to be installed:
openjdk-7-jdk wget upower apcupsd
Setup
You place the compiled .jar file in a handy place along with a few files (most found in the Github repo as examples):
Configuration
index.html # a template HTML file that is used to list all of the servers, uptimes, and other data. server.html # a template HTML file that is used to list one server and all of the associated key and value pairs that it has. master.yml # a file that defines master keys, which are server side keys that define server characteristics locally, used to enable servers, specify if they are urgent to server uptime servers/<servername>.txt # file that configures maintainers
Maintainers Listing (in servers folder)
f:<firstname> l:<lastname> e:<email> i:<irc> c:<cell or other phone number>
You might need to create a servers folder if it crashes.
Inside the servers folder, there are configurable per-server configs.
Make sure that you check that your YAML files are parsed properly or I guarantee that the Java code will crash. There are a few good online checkers out there.
Startup
I made the startup script for the management server much the same as the client one.
The sh file is as follows:
cd /manage date >> runtime.log java -jar management-server.jar >> runtime.txt
Hardware Implementation:
Fetch Armbian Jessie for the pcduino 3. It's OK that it's not the nano lite version even though currently we are using a pcduino 3 nano lite.
Flash that to the SD card, log into the root user set the root password, and then run the reboot command. Wait for it to restart again, and then reboot.
At this point, the system has set up the SSH server, expanded / to the full size of the SD card (up to 32GB), as well as created an emergency 128MB swap partition.
Now, install a thing:
htop openssh-client vim openjdk-7-jdk p7zip-full g++ sudo git upower apcupsd
And now edit some files (make them contain this following contents):
vim /etc/hostname
management
vim /etc/network/interfaces
# Wired adapter #1 auto eth0 iface eth0 inet static address 128.153.145.62 netmask 255.255.254.0 gateway 128.153.145.1 # Local loopback auto lo iface lo inet loopback
and edit the sshd config for the default cosi ssh port:
vim /etc/ssh/sshd_config
set the line that says Port
After you have done that, reboot.
You are now to follow the default instructions for setting up the software itself.
After that, install these:
apache2 mysql-server cacti
Add an SNMP device (ucd/net SNMP Host) that is the network switches.
After that, add graphs that are Bytes Total Bandwidth, for each ethernet port. From there, have fun configuring.
Start Scripts:
System V Start Script
/etc/init.d/manage (the name of the control script will be manage - don't use any extension and make sure it is executable)
### BEGIN INIT INFO # Provides: manage # this needs to match the name of the file # Required-Start: $remote_fs $syslog $network $all # Required-Stop: $remove_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 ### END INIT INFO /usr/bin/java -jar /manage/management-server.jar > runtime.log & # make your executable run in here.
and make that run at startup with:
update-rc.d manage defaults
For more on LSB start scripts, visit https://wiki.debian.org/LSBInitScripts
Systemd Start Script
/etc/systemd/system/manage.service: (the service will be called manage, due to the filename. Always append .service to the service file)
[Unit] Description=manage # use a title for the application that will help someone realize what is going on [Service] ExecStart=/bin/bash /manage/run.sh # point this to the executable you intend to run. #ExecStop= # point this to a stop script if you have one, if you don't it will just kill the process when you tell it to stop [Install] WantedBy=multi-user.target
Systemd Helpful Tips
systemctl enable <name> for enabling units
systemctl disable <name> for disabling
systemctl start <name> for starting
systemctl stop <name> for stopping
systemctl status <name> for the executable status.
Plans:
Additional planned features (in this general order) are:
- thermal information
- graph display of events
- database system to store the data collected
- tell servers when exactly to power down in the event of an outage or over temp situation - alternatively, email maintainers to shut down the servers