Structure of the Cluster¶
This section tells you what each of our servers is and what software it needs to function. Configuration is only discussed in broad strokes, and you won’t find anything on the actual administration of these machines and services; such documentation can (ideally, at least) be found elsewhere in these notes.
![digraph acmsys {
compound=true;
clusterrank="local";
rankdir="LR";
subgraph clusterSunfire0 {
label = "sunfire0";
rank = 0;
Sunfire0_ZFS [ label="ZFS",style=filled,fillcolor="yellow"] ;
Sunfire0_Ceph_OSD [ label="osd.1",style=filled,fillcolor="orange",rank=1] ;
Sunfire0_Ceph_OSD -> Sunfire0_ZFS [constraint=false];
};
subgraph clusterSunfire1 {
label = "sunfire1";
rank = 0;
Sunfire1_ZFS [ label="ZFS",style=filled,fillcolor="yellow"] ;
Sunfire1_Ceph_OSD [ label="osd.2",style=filled,fillcolor="orange"] ;
Sunfire1_Ceph_OSD -> Sunfire1_ZFS [constraint=false];
};
subgraph clusterSunfire2 {
label = "sunfire2";
rank = 0;
Sunfire2_ZFS [ label="ZFS",style=filled,fillcolor="yellow"] ;
Sunfire2_Ceph_OSD [ label="osd.0",style=filled,fillcolor="orange"] ;
Sunfire2_Ceph_OSD -> Sunfire2_ZFS [constraint=false];
};
subgraph clusterSunfire3 {
label = "sunfire3";
rank = 0;
Sunfire3_ZFS [ label="ZFS",style=filled,fillcolor="yellow"] ;
Sunfire3_Ceph_OSD [ label="osd.3",style=filled,fillcolor="orange"] ;
Sunfire3_Ceph_OSD -> Sunfire3_ZFS [constraint=false];
};
subgraph clusterMagellan {
label = "Magellan";
subgraph clusterAfsFile {
label = "AFSFile";
Afsfile_AFS_FS [ label="AFS FS",style=filled,fillcolor="purple" ];
};
subgraph clusterAfsScratch {
label = "AFSScratch";
Afsscratch_AFS_FS [ label="AFS FS",style=filled,fillcolor="purple" ];
};
Magellan_Ceph_Mon [ label="Ceph Mon",style=filled,fillcolor="orange"];
Magellan_Shorewall [ label="Shorewall",style=filled,fillcolor="red"];
};
subgraph clusterCrimea {
label = "Crimea";
Crimea_Ceph_Mon [ label="Ceph Mon",style=filled,fillcolor="orange"];
Crimea_SMTP;
};
subgraph clusterGomes {
label = "Gomes";
Gomes_Ceph_Mon [ label="Ceph Mon",style=filled,fillcolor="orange"];
Gomes_Shorewall [ label="Shorewall", style=filled,fillcolor="red"];
Gomes_Nova [ label="Nova Controller", style=filled ];
};
subgraph clusterChicago {
label = "Chicago";
Chicago_AFS_PTS [ style=filled,fillcolor="green" ];
Chicago_AFS_VL [ style=filled,fillcolor="green" ];
Chicago_AFS_FS [ style=filled,fillcolor="purple" ];
};
subgraph clusterTyphon {
label = "Typhon";
Typhon_AFS_PTS [ style=filled,fillcolor="green" ];
Typhon_AFS_VL [ style=filled,fillcolor="green" ];
Typhon_AFS_FS [ style=filled,fillcolor="purple" ];
};
subgraph clusterEchidna {
label = "Echidna";
Echidna_AFS_PTS [ style=filled,fillcolor="green" ];
Echidna_AFS_VL [ style=filled,fillcolor="green" ];
Echidna_AFS_FS [ style=filled,fillcolor="purple" ];
};
virtual_cephmon [ label="Ceph Mon",style=filled,fillcolor="orange" ];
virtual_AFS_PTS [ label="AFS PTS",style=filled,fillcolor="green" ];
virtual_AFS_VL [ label="AFS VL",style=filled,fillcolor="green" ];
virtual_cephmon -> Magellan_Ceph_Mon;
virtual_cephmon -> Crimea_Ceph_Mon;
virtual_cephmon -> Gomes_Ceph_Mon;
Sunfire0_Ceph_OSD -> virtual_cephmon;
Sunfire1_Ceph_OSD -> virtual_cephmon;
Sunfire2_Ceph_OSD -> virtual_cephmon;
Sunfire3_Ceph_OSD -> virtual_cephmon;
virtual_AFS_VL -> Chicago_AFS_VL;
virtual_AFS_VL -> Echidna_AFS_VL;
virtual_AFS_VL -> Typhon_AFS_VL;
Afsscratch_AFS_FS -> virtual_AFS_VL;
Afsfile_AFS_FS -> virtual_AFS_VL;
Echidna_AFS_FS -> virtual_AFS_VL;
Chicago_AFS_FS -> virtual_AFS_VL;
Typhon_AFS_FS -> virtual_AFS_VL;
Crimea_SMTP -> Afsfile_AFS_FS [ style=dotted] ;
virtual_AFS_PTS -> Chicago_AFS_PTS;
virtual_AFS_PTS -> Echidna_AFS_PTS;
virtual_AFS_PTS -> Typhon_AFS_PTS;
Afsscratch_AFS_FS -> virtual_AFS_PTS;
Afsfile_AFS_FS -> virtual_AFS_PTS;
Echidna_AFS_FS -> virtual_AFS_PTS;
Chicago_AFS_FS -> virtual_AFS_PTS;
Typhon_AFS_FS -> virtual_AFS_PTS;
subgraph clusterGlossary {
Storage [ style=filled,fillcolor="yellow" ];
Ceph [ style=filled,fillcolor="orange" ];
"AFS FS" [ style=filled,fillcolor="purple" ];
"AFS DB" [ style=filled,fillcolor="green" ];
"NAT" [ style=filled,fillcolor="red" ];
};
// subgraph clusterKDC {
// label = "KDC";
// KDC_Master;
// KDC_Replicas -> KDC_Master [ label="kpropd" ];
// };
// subgraph clusterLDAP {
// label = "LDAP";
// LDAP_Master;
// LDAP_Replicas -> LDAP_Master [ label="syncrepl" ];
// LDAP_Replicas -> KDC_Master [ lhead=clusterKDC ] ;
// };
//
// subgraph clusterCeph {
// label = "Ceph";
// subgraph clusterCephMon {
// label = "Mon";
// Ceph_Mon_Magellan;
// Ceph_Mon_Gomes;
// };
// };
}](../../_images/graphviz-d3d42950bb8e758a358e1c72946e5b38f50690e9.png)
Control Layer¶
Much of the cluster is controlled from three Dell 2950 nodes. All three machines are ceph monitors; ceph is therefore critical software on all three. They are otherwise (relatively) simple debian machines. All three machines need afs clients for various bits of configuration. Each machine handles a couple of other services:
Magellan¶
Magellan serves as the primary gateway from the cluster to the wider internet. It runs our firewall via shorewall. Magellan also hosts afs-file and afs-scratch, our virtual afs servers. These run through libvirt. You can generally ssh from magellan to any physical machine in the cluster. (Openstack) VMs are not guaranteed.
Magellan also has some nice aliases for working with ipmitool in localadmin’s
.bashrc.
Gomes¶
Gomes’s main job is running Openstack. It runs nova, keystone, glance, and cinder; these are all parts of Openstack (and, thus, gomes-critical software). Gomes also does networking out of Openstack, so it needs shorewall as well.
Crimea¶
Crimea is our mailserver. It uses [stuff] to do this.
Storage Layer¶
Our storage machines need ceph for replication. They also need afs clients for common configuration. Further, their underlying filesystem is zfs, so they need those packages as well. Currently, our storage layer consists of four sunfires named (appropriately): sunfire0, sunfire1, sunfire2, and sunfire3.
Compute Nodes¶
Our three compute nodes are Antonio, Enrique, and Serrao. They need [Openstack software] to do their jobs. All three are Dell R900s.
Virtual Hosts¶
AFS Virtual Servers¶
AFS is controlled by two virtual machines on Magellan: these are afs-file and afs-scratch. Each does [stuff].
Openstack Service VMs¶
We have a large number of virtual machines of varying importance that live in Openstack under the admin tenant. Here’s a quick rundown:
- Conch and cowrie are simple shell servers. They have most of the same software as our desktops, but without the graphical stuff. Conch sits outside the Hopkins firewall; cowrie is inside it.
- Egg is another shell server, but it’s special. It’s configured to allow login via JHED for people who for whatever reason are using our systems but aren’t ACM members.
- Einstein is our database VM. Most of our other vms store their databases on Einstein.
- Ebola (thanks Rich) is our bugtracker. It runs request-tracker, a perl/mason utility. It’s got the RT “assets” plugin for inventory tracking, but we’ve (obviously) decided to move away from that system.
- Git runs gitlab. Nothing much to see here.
- Ci-runner-1 is our gitlab continuous integration worker machine. We should probably have a couple of public, general workers, but we should really also allow people to set up their own CI runners. This will involve looking into gitlab’s config.
- Astrolabe
- Belthazar
- Quassel is our quasselcore. Ben Rosser has a pull request on quassel that would allow us to use LDAP as an authentication backend, which in turn would let us to give people a quassel account with their ACM account (by allowing LDAP authentication) but it hasn’t been merged into the master yet.
- Sharpsphere
- Web and web-v2 are supposed to be our webservers. They are somewhat screwed up at the moment.
- Wintermute
- Jupyter runs an ipython server.
- Hermes runs not-acmbot, and records our IRC channel.
- Minerva was intended to be a print server, but isn’t fully set up.
- Aperture runs SETI@home and similar crowdsource-computing software.
- Blocks is a minecraft server.
- Cubes is a minetest server.
- There was some thought that the ACM could set up a forum using phpbb or something similar. As with several other ACM projects, this went nowhere, but Gazebo was to be the host for it.
- Knalga is a Wesnoth server.
- Musicbrainz
- Nymph
- Threefort might be a TF2 server?
Chicago¶
Chicago handles our backups. It runs zfs-root (and thus needs those packages), and also does [afs server-side things].
Desktops¶
Our desktops are also shell servers; we encourage people to log in to them because they are generally somewhat faster than our virtual shell servers. These machines need afs clients, ssh servers, plus a whole load of graphical software. Eridani, arcturus, and fomalhaut are currently set up as actual two-monitor desktops. Sirius and polaris are currently running some of the north wall services; they are otherwise normal desktops. Vega, deneb, and procyon are all off; they are state unknown.