Difference between revisions of "Network Setup"

From CSLabsWiki
Jump to: navigation, search
(Portmaps)
 
(54 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
'''THIS IS OUTDATED'''
  
=New Information=
+
='''THIS IS OUTDATED'''=
  
 
==Basics==
 
==Basics==
  
The connection to OIT is provided by a Gigabit CAT5 cable. This CAT5 cable plugs into the OIT switch's inbound Gigabit port (SW0). A patch cable connects the outbound Gigabit port to SWM1. From SWM1, there are 3 branches to ITL switches, branches to Ziltoid (COSI's Firewall), Mirror, Tor-Exit, DDC-Router, and SWX. SWM2 and SWM3 are inside the server racks and distribute network to those racks, except for SWX and DDC which provide network for Grad Machines and the DDC, respectively.
+
The connection to OIT is provided by a Gigabit CAT5e cable. This CAT5e cable plugs into the OIT switch's Gigabit port 2 (sc-344-a). There is also a patch cable that connects port 46 of the 100M ports on sc344 which provides the 128.153.146.0/24. A patch cable connects the outbound Gigabit port 2 to SWM1. From SWM1, there are 2 branches to ITL switches, branches to Ziltoid (COSI's Firewall), Mirror and Tor-Exit. SWM2 and SWM3 are inside the server racks and distribute network to those racks.
  
Behind Ziltoid, the patch cables for all the Ethernet and the WiFi inside COSI itself are connected to a handful of switches.
+
Behind Ziltoid, the patch cables for all the Ethernet and the WiFi inside COSI itself are connected to a handful of switches. On a seperate two ports on Ziltoid there are cables to SWM2 and SWM3
  
The ITL is connected into some 24 and 48 port switches which distribute network directly. A second switch which is normally unplugged is used to give the ITL network internal connectivity for botnet and virus research.
+
The ITL is connected into some 24 and 48 port switches which distribute network directly. A third switch which is normally unplugged is used to give the ITL network internal connectivity for botnet and virus research.
 +
 
 +
== STP ==
 +
 
 +
STP is enabled on all ports on the managed switches, and when a loop is detected, the port will shut down. Once a loop has been resolved, it can take up to 10 seconds before the switch tries again.
 +
 
 +
<b>DO NOT EVER ENABLE STP ON THE OIT UPLINK PORTS!! YOU WILL GET DISCONNECTED!</b>
 +
 
 +
Those ports are <b>port 1</b> and <b>port 19</b> on SWM1.
 +
 
 +
If you do this, submit an OIT RT ticket saying you're sorry, and that you need the port reset, because BPDU guard detected a STP packet. Just one is all it takes.
 +
 
 +
Don't ask them to disable it either, they will get salty (and for good reason).
 +
 
 +
==DHCP/DNS/etc==
 +
 
 +
DHCP, DNS, and LDAP/Kerberos are all handled by Talos
 +
 
 +
Parts of 128.153.144.0/24 are on DHCP
 +
 
 +
All of 128.153.145.0/24 are static allocation (by host)
  
 
==Portmaps==
 
==Portmaps==
  
 
This is a list of ports to which are connected to our managed switches. It is highly advised not to move machines from their location on the switches without permission from someone who knows the per-port configuration on the switch in question, otherwise some machines may not be properly throttled or managed, resulting in (dire) consequences.
 
This is a list of ports to which are connected to our managed switches. It is highly advised not to move machines from their location on the switches without permission from someone who knows the per-port configuration on the switch in question, otherwise some machines may not be properly throttled or managed, resulting in (dire) consequences.
 +
 +
Upload and Download limits are machine centric. The internal switch dialogs are switch centric, thus "ingress" from switch to machine is upload and "egress" from switch to machine is download.
  
 
===SWM1 Portmap===
 
===SWM1 Portmap===
 +
 +
<b>Ports 1 and 19 should NEVER see STP or you will be sad. NEVER CHANGE UPLINK PORTS!</b>
  
 
{| class="mw-collapsible mw-collapsed wikitable"
 
{| class="mw-collapsible mw-collapsed wikitable"
! Port Number
+
! Port
 
! Allocation
 
! Allocation
 +
! VLAN
 +
! Upload Kb/s
 +
! Download Kb/s
 
|-
 
|-
 
|1
 
|1
|OIT Link
+
|OIT Link (sc-344-a, port 2)
 +
|1, Access
 +
|
 +
|
 
|-
 
|-
 
|2
 
|2
 +
|DEBUG
 +
|1, Access
 +
|
 
|
 
|
 
|-
 
|-
 
|4
 
|4
 +
|Ziltoid
 +
|1, Access
 +
|
 
|
 
|
 
|-
 
|-
 
|5
 
|5
 +
|ITL SW6
 +
|1, Access
 
|
 
|
|-
 
|6
 
 
|
 
|
 
|-
 
|-
 
|7
 
|7
 +
|ITL SW5
 +
|1, Access
 
|
 
|
|-
 
|9
 
 
|
 
|
 
|-
 
|-
 
|10
 
|10
 +
|Mirror
 +
|1, Access
 +
|750016
 +
|299968
 +
|-
 +
|17
 +
|NC
 +
|2, Access
 +
|
 
|
 
|
 
|-
 
|-
|11
+
|18
 +
|NC
 +
|2, Access
 +
|
 
|
 
|
 
|-
 
|-
|12
+
|19
 +
|OIT Uplink (sc-334-a, port 46)
 +
|2, Access
 +
|
 
|
 
|
 
|-
 
|-
|14
+
|20
 +
|Tor-Exit
 +
|2, Access
 +
|
 
|
 
|
 
|}
 
|}
Line 57: Line 112:
  
 
{| class="mw-collapsible mw-collapsed wikitable"
 
{| class="mw-collapsible mw-collapsed wikitable"
! Port Number
+
! Port
 
! Allocation
 
! Allocation
 +
! Upload Kb/s
 +
! Download Kb/s
 
|-
 
|-
 
|1
 
|1
 +
|Ziltoid
 +
|
 
|
 
|
 
|-
 
|-
 
|2
 
|2
 +
|DEBUGGING
 +
|
 
|
 
|
 
|-
 
|-
 
|3
 
|3
 +
|SWM3
 +
|
 
|
 
|
 
|-
 
|-
 
|4
 
|4
 +
|Bacon
 
|
 
|
|-
 
|5
 
 
|
 
|
 
|-
 
|-
 
|6
 
|6
 +
|Bennu
 
|
 
|
|-
 
|7
 
|
 
|-
 
|8
 
|
 
|-
 
|9
 
 
|
 
|
 
|-
 
|-
 
|10
 
|10
 +
|The-Internet
 
|
 
|
|-
 
|11
 
 
|
 
|
 
|-
 
|-
 
|12
 
|12
 +
|NAS1
 
|
 
|
|-
 
|13
 
 
|
 
|
 
|-
 
|-
 
|14
 
|14
 +
|Talos
 
|
 
|
|-
 
|15
 
|
 
|-
 
|16
 
 
|
 
|
 
|-
 
|-
 
|17
 
|17
 +
|Red Dwarf
 
|
 
|
|-
+
|99968
|18
 
|
 
 
|-
 
|-
 
|19
 
|19
 +
|Prometheus
 
|
 
|
|-
+
|99968
|20
 
|
 
|-
 
|21
 
|
 
|-
 
|22
 
|
 
|-
 
|23
 
|
 
|-
 
|24
 
|
 
 
|}
 
|}
  
 
===SWM3 Portmap===
 
===SWM3 Portmap===
 
 
{| class="mw-collapsible mw-collapsed wikitable"
 
{| class="mw-collapsible mw-collapsed wikitable"
! Port Number
+
! Port
 
! Allocation
 
! Allocation
 +
! Upload Kb/s
 +
! Download Kb/s
 
|-
 
|-
 
|1
 
|1
 +
|Ziltoid
 +
|
 
|
 
|
 
|-
 
|-
 
|2
 
|2
 +
|DEBUGGING
 +
|
 
|
 
|
 
|-
 
|-
 
|3
 
|3
 +
|SWM2
 
|
 
|
|-
 
|4
 
|
 
|-
 
|5
 
 
|
 
|
 
|-
 
|-
 
|6
 
|6
 +
|Elephant
 
|
 
|
|-
 
|7
 
 
|
 
|
 
|-
 
|-
|8
+
|10
 +
|Gitlab
 
|
 
|
|-
 
|9
 
 
|
 
|
 
|-
 
|-
|10
+
|11
 +
|Hydra@ETH1
 
|
 
|
|-
 
|11
 
 
|
 
|
 
|-
 
|-
 
|12
 
|12
 +
|Hydra@ETH0
 +
|
 
|
 
|
 
|-
 
|-
 
|13
 
|13
 +
|Grand-Dad@ETH0
 +
|
 
|
 
|
 
|-
 
|-
 
|14
 
|14
 +
|Grand-Dad@ETH1
 
|
 
|
|-
 
|15
 
 
|
 
|
 
|-
 
|-
 
|16
 
|16
 +
|Androbattery
 
|
 
|
|-
 
|17
 
 
|
 
|
 
|-
 
|-
|18
+
|21
|
+
|Management-Monitor
|-
 
|19
 
 
|
 
|
|-
 
|20
 
|
 
|-
 
|21
 
 
|
 
|
 
|-
 
|-
 
|22
 
|22
 +
|Norm
 
|
 
|
|-
 
|23
 
|
 
|-
 
|24
 
 
|
 
|
 
|}
 
|}
  
=Old Information=
+
===On-Switch Portmaps===
 
 
==Main Ideas==
 
The cslab's networking equipment is split into several categories: External, COSI Internal, ITL Internal, and Server Room Internal.  In the spring of 2011, the wiring organization and layout of the lab's network setup was redesigned by Christian Mesh, who was advised by Mathew S. McCarrell.
 
 
 
==External Network==
 
This network allows the cslabs to connect to the outside world via OIT's infrastructure.
 
 
 
===Basics===
 
The connection to OIT is provided via a Gigibit CAT 5 cable.  That CAT 5 cable plugs into OIT switch 1's first Gigabit port.  A patch cable connects OIT switch 1's second Gigabit port to OIT switch 2's first port.  OIT switch 2's port 2 is then used to provide a Gigabit connection to Mirror.  In addition, cslab's external switch 1 & 2 are connected to OIT switch 2's port 21 and 18, respectively.
 
 
 
===DHCP/DNS===
 
[http://www.clarkson.edu/oit/ OIT] provides dynamic DHCP/DNS to the labs on the 128.153.144.* subnet.  All machines using DHCP in the labs receive a host name of <name>.sclab.clarkson.edu.  We also have some static IPs reserved on the 128.153.144.* subnet for our lab PCs to use.
 
 
 
See [[List of Static IPs - 144 Subnet]] for more information on the static IPs used on the 128.153.144.* subnet.
 
See [[List of Static IPs - 145 Subnet]] for more information on the static IPs used on the 128.153.145.* subnet.
 
 
 
 
 
==COSI Internal Network==
 
COSI's internal network is used to connect all of the lab PCs and most of our servers to an isolated Gigabit network. This network is primarily used for projects and cloning the lab.
 
 
 
===DNS===
 
DNS is provided for the internal network by [[Dns1]] & [[Dns2]].  All DNS entries are <name>.int.cslabs.clarkson.edu.
 
 
 
See [List of Static IPs - Internal_Network] for more information
 
 
 
 
 
==ITL Internal Network==
 
ITL's internal network is used to connect all of the lab PCs to an isolated Gigabit network. This network is primarily used for projects, class demonstrations, and cloning the lab.  The network consists of 2 Gigabit switches.  Switch 1 has ITL 1 to 12 on ports 1 to 12, a cable to COSI's internal network on port 13, and a patch cable to the switch 2 on port 24.  Switch 2 has ITL 13 to 24 on ports 13 to 24, ITL 25 on port 5, ITL TS on port 1, ITL GS on port 23, and the patch cable to switch 1 on port 12.
 
  
===Port Layout===
+
All of our managed switches contain descriptions of the port; after logging in, proceed to Switching > Port, and, under Port Config (the default pane), you will see the Description column. The data in those columns should correspond to what you see above; if not, please consult the physical status of the network and update both accordingly.
[[Image:ITLINTSW.jpg|500px]]
 
  
===DNS===
+
The Description field stored by our TP-Link switches is only 16 characters wide and can contain only alphanumerics and the characters "@-_:/.", which significantly limits the usability of this field. Nonetheless, I've come up with the following schema for the field:
See [[List of Static IPs - Internal Network]] for more information
 
  
 +
:'''''type''':'''name'''@'''ident'''''
  
==Server Internal Network==
+
The ''type'' field may contain any of the following:
This network is used to connect COSI's servers with a secure Gigabit connection.
 
  
===DNS===
+
* '''UL''' indicates an ''uplink''--a hop to a router with a smaller prefix (more general routing) or closer to the center of the hub in a hub-spoke network.
DNS is provided for the server room network by [[Dns1]] & [[Dns2]]. All DNS entries are <name>.sr.cslabs.clarkson.edu.
+
* '''DL''' indicates a ''downlink''--a hop to a router with a larger prefix (toward a host) or closer to the edges of a hub-spoke network.
 +
* '''XL''' indicates a ''crosslink''--a hop between routers with the same prefix, often included for redundancy.
 +
* '''UL:H''', '''DL:H''', and '''XL:H''' indicate ''hosts''--not routers or switches per se, but general-purpose computers--that are responsible for forwarding traffic as an uplink, downlink, or crosslink, respectively. Firewalls (such as [[Ziltoid]]) and host-based NATs (such as [[DDC-router]]) fall into this category.
 +
* '''H''' is a ''host''--a general purpose computer. Most of our services, including repositories ([[Mirror]]), VM hosts ([[Bennu]], [[Hydra]]), tunnelling endpoints [[Tor-exit]], and general services ([[Androbattery]]) fall into this category. At present, this scheme only considers ''physical'' hosts, and so it does not count, e.g., VMs which are responsible for routing--this may change if there is a need for it.
 +
* '''DEBUG''' is an open port reserved for debugging the switch--it should always have access to the switch's management interface. Such ports generally have no name.
  
 +
Not all uplinks and downlinks are reciprocated in this configuration; for example, the "downlinked" switch may not be managed, and thus not be able to have port descriptions.
  
==Patch Panels==
+
The ''name'' field ''should'' contain the canonical name of the ''remote end'' of that port's cable; for switches, this is usually named "SW*" (e.g., [[SWM1]], [[SW0]], etc.), and for hosts, this is either its [[DNS]] name or its self-recognized hostname. If there exists no sensible name for this connection, the name may be elided (along with the colon :), but ''only'' if the ''ident'' field may also be elided. An empty name is permitted in the latter circumstance.
The patch panels provide an easy way to tie ports around the room into different switches and keep track of them easily.
 
===COSI===
 
explanation
 
===ITL===
 
explanation
 
  
==Wiring Idiosyncrasies==
+
The ''ident'' name should be a non-negative integer that can disambiguate multiple links between the same switch/host. The order of numbering is intentionally left to the implementer, who may base it on interface indices, port numbers, etc.. In many cases, it can be omitted, along with the at-sign (@). Current implementations, for example, use the ident field to refer to different ports on hosts participating in a LAG.
There are several small organizational ideas that were implemented to make changing and following cables easier:
 
  
1. Patch cables in the front of the black ducts, server and misc cables in the back.
+
This standard is living and open to change at any time; perhaps it can be expanded if our switches change.
  
2. The internal networks use green cable and the external network uses blue.  There are several exceptions that should eventually replaced.
+
==Network Statistics==
  
3. Groups of patch cables are generally bundled together because they are near each other on the patch panel.
+
Network statistics are collected by [[Management]] and are printed conveniently on STAT.
  
 +
Visit http://stat.cosi.clarkson.edu/cacti (with default username and password for the labs) and then jump around the tree
  
==Layout==
+
==Network Map==
This is a basic layout of where the switches are located in the two server room racks
 
  
[[Image:NetworkReorg.jpg|400px]]
+
[[Image:Cosi-network.svg|600px]]
  
[[Category:Infastructure]]
+
[[Category:infrastructure]]

Latest revision as of 17:17, 21 May 2018

THIS IS OUTDATED

THIS IS OUTDATED

Basics

The connection to OIT is provided by a Gigabit CAT5e cable. This CAT5e cable plugs into the OIT switch's Gigabit port 2 (sc-344-a). There is also a patch cable that connects port 46 of the 100M ports on sc344 which provides the 128.153.146.0/24. A patch cable connects the outbound Gigabit port 2 to SWM1. From SWM1, there are 2 branches to ITL switches, branches to Ziltoid (COSI's Firewall), Mirror and Tor-Exit. SWM2 and SWM3 are inside the server racks and distribute network to those racks.

Behind Ziltoid, the patch cables for all the Ethernet and the WiFi inside COSI itself are connected to a handful of switches. On a seperate two ports on Ziltoid there are cables to SWM2 and SWM3

The ITL is connected into some 24 and 48 port switches which distribute network directly. A third switch which is normally unplugged is used to give the ITL network internal connectivity for botnet and virus research.

STP

STP is enabled on all ports on the managed switches, and when a loop is detected, the port will shut down. Once a loop has been resolved, it can take up to 10 seconds before the switch tries again.

DO NOT EVER ENABLE STP ON THE OIT UPLINK PORTS!! YOU WILL GET DISCONNECTED!

Those ports are port 1 and port 19 on SWM1.

If you do this, submit an OIT RT ticket saying you're sorry, and that you need the port reset, because BPDU guard detected a STP packet. Just one is all it takes.

Don't ask them to disable it either, they will get salty (and for good reason).

DHCP/DNS/etc

DHCP, DNS, and LDAP/Kerberos are all handled by Talos

Parts of 128.153.144.0/24 are on DHCP

All of 128.153.145.0/24 are static allocation (by host)

Portmaps

This is a list of ports to which are connected to our managed switches. It is highly advised not to move machines from their location on the switches without permission from someone who knows the per-port configuration on the switch in question, otherwise some machines may not be properly throttled or managed, resulting in (dire) consequences.

Upload and Download limits are machine centric. The internal switch dialogs are switch centric, thus "ingress" from switch to machine is upload and "egress" from switch to machine is download.

SWM1 Portmap

Ports 1 and 19 should NEVER see STP or you will be sad. NEVER CHANGE UPLINK PORTS!

Port Allocation VLAN Upload Kb/s Download Kb/s
1 OIT Link (sc-344-a, port 2) 1, Access
2 DEBUG 1, Access
4 Ziltoid 1, Access
5 ITL SW6 1, Access
7 ITL SW5 1, Access
10 Mirror 1, Access 750016 299968
17 NC 2, Access
18 NC 2, Access
19 OIT Uplink (sc-334-a, port 46) 2, Access
20 Tor-Exit 2, Access

SWM2 Portmap

Port Allocation Upload Kb/s Download Kb/s
1 Ziltoid
2 DEBUGGING
3 SWM3
4 Bacon
6 Bennu
10 The-Internet
12 NAS1
14 Talos
17 Red Dwarf 99968
19 Prometheus 99968

SWM3 Portmap

Port Allocation Upload Kb/s Download Kb/s
1 Ziltoid
2 DEBUGGING
3 SWM2
6 Elephant
10 Gitlab
11 Hydra@ETH1
12 Hydra@ETH0
13 Grand-Dad@ETH0
14 Grand-Dad@ETH1
16 Androbattery
21 Management-Monitor
22 Norm

On-Switch Portmaps

All of our managed switches contain descriptions of the port; after logging in, proceed to Switching > Port, and, under Port Config (the default pane), you will see the Description column. The data in those columns should correspond to what you see above; if not, please consult the physical status of the network and update both accordingly.

The Description field stored by our TP-Link switches is only 16 characters wide and can contain only alphanumerics and the characters "@-_:/.", which significantly limits the usability of this field. Nonetheless, I've come up with the following schema for the field:

type:name@ident

The type field may contain any of the following:

  • UL indicates an uplink--a hop to a router with a smaller prefix (more general routing) or closer to the center of the hub in a hub-spoke network.
  • DL indicates a downlink--a hop to a router with a larger prefix (toward a host) or closer to the edges of a hub-spoke network.
  • XL indicates a crosslink--a hop between routers with the same prefix, often included for redundancy.
  • UL:H, DL:H, and XL:H indicate hosts--not routers or switches per se, but general-purpose computers--that are responsible for forwarding traffic as an uplink, downlink, or crosslink, respectively. Firewalls (such as Ziltoid) and host-based NATs (such as DDC-router) fall into this category.
  • H is a host--a general purpose computer. Most of our services, including repositories (Mirror), VM hosts (Bennu, Hydra), tunnelling endpoints Tor-exit, and general services (Androbattery) fall into this category. At present, this scheme only considers physical hosts, and so it does not count, e.g., VMs which are responsible for routing--this may change if there is a need for it.
  • DEBUG is an open port reserved for debugging the switch--it should always have access to the switch's management interface. Such ports generally have no name.

Not all uplinks and downlinks are reciprocated in this configuration; for example, the "downlinked" switch may not be managed, and thus not be able to have port descriptions.

The name field should contain the canonical name of the remote end of that port's cable; for switches, this is usually named "SW*" (e.g., SWM1, SW0, etc.), and for hosts, this is either its DNS name or its self-recognized hostname. If there exists no sensible name for this connection, the name may be elided (along with the colon :), but only if the ident field may also be elided. An empty name is permitted in the latter circumstance.

The ident name should be a non-negative integer that can disambiguate multiple links between the same switch/host. The order of numbering is intentionally left to the implementer, who may base it on interface indices, port numbers, etc.. In many cases, it can be omitted, along with the at-sign (@). Current implementations, for example, use the ident field to refer to different ports on hosts participating in a LAG.

This standard is living and open to change at any time; perhaps it can be expanded if our switches change.

Network Statistics

Network statistics are collected by Management and are printed conveniently on STAT.

Visit http://stat.cosi.clarkson.edu/cacti (with default username and password for the labs) and then jump around the tree

Network Map

Cosi-network.svg