External Network Considerations¶
Allocations¶
We’ve got a lot of IP addresses and ranges allocated to us by various parties. Here’s an attempt to keep track of them all.
| Address/block | Where | What | From Whom |
|---|---|---|---|
| 128.220.70.0/24 | Malone VLAN 13 | Cluster JHU Internal (“oldcs”) | JHU IT |
| 128.220.251.32/29 | Malone DMZ | Cluster JHU DMZ (“ff”) | JHU IT |
| 128.220.35.176/28 | Malone VLAN 35 | CS public subnet | CS |
| 10.161.159.216/29 | Malone VLAN 159 | CS private subnet | CS |
| 2606:2B00:0:410::/64 | ??? | JHU IPv6 network | JHU IT |
Internally, the subnets from JHU IT are allocated to:
| Address/block | Controller | What |
|---|---|---|
| 128.220.70.0/25 | Various | ACM servies and physical hosts |
| 128.220.70.128/26 | Gomes via Openstack | ACM virtual machines |
| 128.220.70.192/26 | Gomes via OpenStack | User virtual machines |
| 128.220.251.32/29 | Magellan, Gomes | See tables below |
Note
As of this writing, 251.38 is unallocated.
Security Policies¶
Network security manages the JHU border gateway policy for 128.220.70.0/24
and requires us to have a clean-slate report to their scanning tools for
external access to be granted. Contact network.security@jhu.edu to get the
policy adjusted, but please try to keep the tables below up-to-date, too!
This means, among other things, that we are obligated to not attempt
IP-address-based restrictions that would keep the following IP addresses from
probing our systems: 10.181.169.162, 10.181.169.163,
10.181.169.164, 10.15.69.217, 10.15.69.218, 10.15.69.219,
128.220.242.60, 10.131.228.26, and 10.132.160.55.
Thankfully, for the most part, public services offered by our cluster are not
restricted by IP address anyway.
Naming¶
Details of how subnet DNS entries are managed can be found in Managing External DNS.
DHCP or other Dynamic Configuration¶
Our allocations from CS can be managed by CS’s DHCP server; for somewhat
obvious reasons they don’t want us running our own on their network. To adjust
the MAC/IP map, send mail to support@cs.
Our direct allocations are manually managed and do not use dynamic configuration.
Cluster Common Considerations¶
The cluster is a big mess internally that gets services exposed on a handful of IP addresses, both inside the JHU firewall and outside and has somewhat interesting egress rules. This page attempts to document the thinking behind some of our port maps, but is non-authoritative (the authority, of course, is what is configured on the cluster gateway).
For the moment, we use shorewall to manage our network configuration.
Multi-Provider Egress and Tracking¶
We have two providers configured, in /etc/shorewall/providers:
csprov 1 0x1000 main $NET_IF_CS 128.220.70.1 track $NET_IFS_INTERNAL
ffprov 2 0x2000 main $NET_IF_FF 128.220.251.33 track $NET_IFS_INTERNAL
The track directive ensures that we route responses back out the interface
on which things arrived. /etc/shorewall/rtrules describes the egress
rules. This file differs between Magellan and Gomes, but in rough schematic:
#SOURCE DEST PROVIDER PRIORITY
- 10.0.0.0/8 csprov 26000
$NET_CIDR_OS_A_CS - csprov 26000
$NET_CIDR_OS_A_FF - ffprov 26000
$NET_CIDR_OS_U_CS - csprov 26000
These rules ensure that, unless otherwise indicated by the ingress-attached
tracking labels, that outbound traffic to JHU-internal RFC1918 addresses
egress via the behind-firewall interface. The $NET_CIDR_OS lines
dictate how egress from our OpenStack VMs is routed – the _A_ regions
are for VMs under administrative control while _U_ are for VMs running
user code.
The contents of /etc/shorewall/masq follow along. Again, this file differs
between Magellan and Gomes, but in rough sketch:
$NET_CS_IF $NET_OS_A_CS_CIDR
$NET_FF_IF $NET_OS_A_FF_CIDR
$NET_CS_IF $NET_OS_U_CS_CIDR $AEOLUS_CS_EXT
Ingress¶
It will probably be clearer to present the contents of
/etc/shorewall/rules in a tabular form:
Magellan¶
| IP Address | Port | JHU Public | Description |
|---|---|---|---|
| .70.63 (magellan) | |||
| TCP 22 | No | Magellan itself listening on SSH | |
| UDP 7000,7005 | Yes | Alt. address for 128.220.251.36 file server | |
| .70.64 (magellan2) | |||
| UDP 7000,7005 | Yes | Alt. address for 128.220.251.35 file server | |
| .251.34 (seattle) | DMZ | ||
| (User firewall free egress address: Multi-Provider Egress and Tracking | |||
| UDP 7000,7005 | AFS scratch and mirror server | ||
| .251.36 (magellan) | DMZ | ||
| TCP 22 | Magellan itself listening on SSH | ||
| UDP 7000,7005 | AFS homedirs and services server |
Gomes¶
| IP Address | Port | JHU Public | Description |
|---|---|---|---|
| .70.55 (astrolabe) | |||
| TCP 80, 443 | Yes | Mirrors web server | |
| .70.65 (centaur) [enet-acm] | |||
| TCP 22 | Yes | All-users SSH server (conch) | |
| TCP 80, 443 | Yes | User web server (web.vm) | |
| TCP 25, 465, 587 | Yes | ACM mail service (centaur.vm) | |
| .70.74 (nagios) | |||
| TCP 80 | Yes | Nagios worker machine (bigbrother.trinidad) | |
| ICMP | Nagios worker machine (bigbrother.trinidad) | ||
| .70.79 | |||
| .70.82 (belthazar) | |||
| TCP 22 | Yes | Egg Shell (JHED AD Integration) | |
| TCP 80,443 | Yes | Mailman web interface (lists.acm.jhu.edu) | |
| (User firewalled egress address: Multi-Provider Egress and Tracking | |||
| .70.84 | |||
| .70.90 | |||
| .70.91 | |||
| .251.35 (batman) | DMZ | ||
| TCP 22 | All-users SSH server (conch.ff.uvm) | ||
| TCP 4242 | Quassel IRC agent (quassel.vm) | ||
| TCP 6080 | Sandstorm alias | ||
| .251.37 (london) | DMZ |
| [enet-acm] | For historical reasons, we have an A record in DNS for our domain. This IP address should probably have the “canonically ACM” things listening on it. At present, this address is inside the JHU firewall. |
Todo
It might be nice to have this table generated automatically from the contents of the various rules files, actually. No?
Services Without the Cluster¶
For the sake of eliminating SPOFs on critical services, the following services are run on hosts entirely outside the cluster gateway. So even if everything falls over, authentication and name resolution should continue to function.
| IP Address | Port | JHU Public | Description |
|---|---|---|---|
| .70.76 (typhon) | |||
| TCP 22 | Yes | SSH | |
| TCP 53 | Yes | DNS | |
| TCP 389 | Yes | LDAP | |
| UDP 53 | Yes | DNS | |
| UDP 88 | Yes | Kerberos KDC | |
| UDP 7000 | Yes | AFS Fileserver (esp. replicas) | |
| UDP 7002,7003 | Yes | AFS DBs | |
| UDP 7005 | No | AFS VolSer | |
| UDP 7007 | No | AFS BosServer | |
| .70.53 (crimea) | Mail server | ||
| TCP 22 | Yes | SSH | |
| TCP 25 | Yes | Mail ingress | |
| TCP 80/443 | Yes | Mailing list web interface | |
| UDP 7000 | Yes | AFS Fileserver reservation Crimea is not an AFS server now! | |
| .35.178 (echidna) | (Services here are replicas from Typhon) | ||
| TCP 22 | Yes | SSH | |
| TCP 53 | Yes | DNS | |
| TCP 389 | Yes | LDAP | |
| UDP 53 | Yes | DNS | |
| UDP 88 | Yes | Kerberos KDC | |
| UDP 7000 | Yes | AFS Fileserver (esp. replicas) | |
| UDP 7002,7003 | Yes | AFS DBs | |
| UDP 7005 | No | AFS VolSer | |
| UDP 7007 | No | AFS BosServer | |
| .35.191 (chicago) | (Most services here are replicas from Typhon) | ||
| TCP 22 | Yes | SSH | |
| TCP 53 | Yes | DNS | |
| TCP 389 | Yes | LDAP | |
| UDP 53 | Yes | DNS | |
| UDP 88 | Yes | Kerberos KDC | |
| UDP 7000 | Yes | AFS Fileserver (esp. replicas) | |
| UDP 7002,7003 | Yes | AFS DBs | |
| UDP 7005 | No | AFS VolSer | |
| UDP 7007 | No | AFS BosServer |
Cluster Uplink Cabling¶
You may also wish to refer to Cluster Switch Cabling for the inside job.
| Host and port | Neighbor |
|---|---|
| Gomes leftmost (eth0) | oldcs (70) |
| Magellan leftmost (eth0) | oldcs (70) |
| Magellan next leftmost (eth1) | DMZ |
| Crimea leftmost | oldcs (70) |
| Chicago eth1 | oldcs (70) |
| Typhon eth0 | oldcs (70) |
| Echidna | cs public (35) |