LDAP

The machines get their user metadata from LDAP, which is on typhon (replicated onto chicago). LDAP isn’t used for authentication anymore [1]; it’s just for metadata now.

[1]Sadly, this is not entirely true yet. See the Non-SASL Auth section for more information on this sadness.

Note that our entire LDAP configuration is public! You should be able to run ldapsearch -x -H ldap://ldap.acm.jhu.edu -b cn=config and see the whole thing. The bits below are non-authoritative but are broken out for exposition.

Data in LDAP

Each user has an entry in LDAP which has their name, login shell, homedir location, uid number, email, and door swipe code. Admins have an extra object (which is maybe not great…) that represents their admin hat and gives them the credentials needed to administer LDAP itself.

Poking at LDAP

Here are the incantations you need to know:

ldappoke -h typhon.acm.jhu.edu ...     # ordinary access (GSSAPI used)
ldappoke -H ldapi:/// -Y EXTERNAL ...   # as local root to do anything

(Substitute the desired command for ldappoke, of course.)

You probably care the most about ldapsearch to see what’s going on, and ldapadd or ldapmodify to change things. Making changes with these tools will involve writing LDIF; it’s pretty intuitive, but see man ldif to check your work.

The low-level LDAP tools get tedious real quick (especially when you have to write LDIF), so if you’ll be poking at LDAP a lot, I recommend looking into ldapvi, a program that fetches LDAP objects, shows them to you in an editor, creates an LDIF representation of any changes you make (which can include adding, deleting, and renaming objects, not just playing with attributes), and applies it. Get tickets for your /admin hat, then:

# Fetch and allow editing of ALL THE THINGS!
ldapvi --discover -h typhon.acm.jhu.edu -Y GSSAPI
# Likewise for the on-line configuration (be careful!)
ldapvi -b cn=config -h typhon.acm.jhu.edu -Y GSSAPI
# Exercise local root's powers, in case things have fallen over.
ldapvi -b cn=config -h ldapi:/// -Y EXTERNAL

(You can add filters and such to these to restrict them to operating on matching objects. See the man page or –help output; saying that there is a wealth of options is an understatement.)

To apply an LDIF file to our configuration, run, with an admin hat on:

ldapmodify -h typhon.acm.jhu.edu < file.ldif

You must have libsasl2-modules-gssapi-mit for the LDAP tools to be able to do GSSAPI auth, which is something that you kind of need.

All sensitive operations and data are only available to users who have authenticated in some capacity, which will be either by GSSAPI (which also encrypts the connection) or over a Unix-domain socket (which means no network is involved at all), so there is no absolute need for TLS, though the servers do support it. Look farther down for more info about that.

Setting up an LDAP Server

We assume you’ve got a full database dump already; if you’re starting afresh, Non-Default Bits of LDAP Configuration may be useful to you. This section assumes a Debian host.

Installing Software

You’ll want to run

apt-get install slapd libsasl2-2 sasl2-bin libsasl2-modules-gssapi-mit
usermod --append --groups sasl

You may also wish to run

apt-get install ldap-utils ldapvi

Setting the Right Host Address

Ensure that /etc/hosts has the host name both as 127.0.1.1 and as its public address. For example:

127.0.1.1     typhon.acm.jhu.edu typhon
128.220.70.76 typhon.acm.jhu.edu

Slapd listens on all addresses returned, but if you set SLAPD_SERVICES in /etc/default/slapd as you might expect to ldapi:/// ldap://${HOSTNAME}, and do not do the above, you’ll end up listening only on the local interface, which is almost assuredly not what you had in mind.

Landing a Keytab

Extract a keytab as usual for the ldap/${HOSTNAME}@ACM.JHU.EDU principal and land it at /etc/ldap/ldap.keytab, mode bits 0440, owned by the openldap user! In /etc/default/slapd add the lines:

export KRB5_KTNAME="FILE:/etc/ldap/ldap.keytab"
export KRB5_CLIENT_KTNAME="FILE:/etc/ldap/ldap.keytab"

The latter one is not listed by default but is vital for our syncrepl setup! Alternatively, you can use /etc/krb5/user/`id -u openldap`/client.keytab for the filename (note backtick expansion), which is the modern multi-UID default for keytabs; if you use it, use it for both KRB5_KTNAME and KRB5_CLIENT_KTNAME.

Creating a jhuacmKerberosInstance DN for the Replica

Be sure to create a DN so that replication can bind appropriately

dn: cn=${HOSTNAME},uid=ldap,ou=Daemons,dc=acm,dc=jhu,dc=edu
uid: ldap
cn: ${HOSTNAME}
objectClass: jhuacmKerberosInstance

Adjusting and Landing the Configuration

You’ll want to adjust the running configuration on the LDAP master to create a new server identifier for replication, then dump the configuration database to a file; you may as well use the official backup /afs/acm.jhu.edu/service/ldap/ldap-config. On the new replica, then:

/etc/init.d/slapd stop
rm -rf /etc/ldap/slapd.d/cn\=config* /var/lib/ldap/*
su openldap -s /bin/bash -c "/usr/sbin/slapadd -c -F /etc/ldap/slapd.d -n 0 -v" \
  < /afs/acm.jhu.edu/service/ldap/ldap-config
/etc/init.d/slapd start

You might also need to adjust /etc/ldap/ldap.conf to set BASE and URI:

BASE    dc=acm,dc=jhu,dc=edu
URI     ldap://ldap.acm.jhu.edu

Non-SASL Auth

Due to a small number of services that have not yet been made to take part in the wonderful world of Kerberos (most notably our rt instance), LDAP is configured to allow simple (non-SASL) password auth against our LDAP user objects. We really should do our best to fix this (i.e. kill it with fire the moment nothing else we care about needs it anymore), as it involves plaintext passwords being handed to slapd.

The userPassword attribute values instruct slapd to verify passwords by doing the client side of SASL. The LDAP servers have saslauthds running to check the passwords against Kerberos when slapd asks. The slapd/saslauthd communication is set up by the following in /etc/ldap/sasl2/slapd.conf:

# Check simple-auth passwords using saslauthd, which in turn uses krb5.
pwcheck_method: saslauthd
saslauthd_path: /var/run/saslauthd/mux

saslauthd itself is mostly configured by /etc/default/saslauthd, so adjust:

START=yes
MECHANISMS=kerberos5

By default, the saslauthd mux is guarded by group membership in the system group sasl; place the openldap user therein.

If you do not have a host/ principal in /etc/krb5.keytab on the server, /etc/saslauthd.conf shoud be used to override which principal is used to verify the KDC, by containing something like:

krb5_verify_principal: ldap

This causes the ldap/ principal’s keytab to be used for this process instead of host/, which, not being present, makes the process fail. ldap/ works just as well.

Note

Whenever the need for this goes away, the userPassword attributes on our user objects should be able to go with it.

Note

It may, as usual, be necessary to wrangle /etc/krb5.conf to fiddle with the rdns option, if saslauthd has trouble getting getting tickets, as it will attempt to construct a principal for verifying the KDC based on the idea of the canonical name of the local host.

Non-Default Bits of LDAP Configuration

Kerberos and GSSAPI

We have kerberized our LDAP configuration. The only truly exciting thing about this is the following attribute:

dn: cn=config
olcAuthzRegexp: {0}^uid=([^,@/]+)/([^,@/]+)(@acm.jhu.edu|),cn=gss(api|-spnego),cn=auth$
  ldap:///dc=acm,dc=jhu,dc=edu??sub?(&(cn=$2)(uid=$1)(objectClass=jhuacmKerberosInstance))
olcAuthzRegexp: {1}^uid=([^,@/]+)(@acm.jhu.edu|),cn=gss(api|-spnego),cn=auth$
  ldap:///dc=acm,dc=jhu,dc=edu??sub?(&(uid=$1)(objectClass=posixAccount))

This works with all GSS libraries we’ve seen so far and is responsible for mapping the GSSAPI presented name to an LDAP DN. Thrilling, isn’t it? It basically takes all our foo/bar@ACM.JHU.EDU Kerberos names and mangles them a bit:

  • Things with an instance get mapped to a jhuacmKerberosInstance.
  • Things without an instance get mapped to a posixAccount.

Warning

slapd as of this writing (and since at least 2009) requires a restart after modifying olcAuthzRegexp. This is undocumented, but a bug was filed and ignored: http://www.openldap.org/lists/openldap-bugs/200903/msg00224.html . So much for “on-line configuration”.

The following stanza in /etc/ldap/sasl2/slapd.conf restricts the offered SASL mechanisms to only GSSAPI and, for ldapi:///, peercred:

mech_list: GSSAPI EXTERNAL

Access Control

The cn=config database has one ACL entry:

dn: olcDatabase={0}config,cn=config
olcAccess: {0}to *
  by dn.regex="^cn=admin,uid=[^,]+,ou=People,dc=acm,dc=jhu,dc=edu$" manage
  by dn.regex="^cn=[^,]+,uid=ldap,ou=Daemons,dc=acm,dc=jhu,dc=edu$" read
  by * read

This allows all */admin hats to do anything, all ldap/* hats to read everything (which will be used for replication), allows everyone to read (broken out for future compatibility), and denies everything else. There is no longer any kind of in-LDAP superuser - just put your admin hat on to acquire Real Ultimate Power(tm), as is the way such things usually go.

The other database, with user entries, is similar but more verbose.

  • As for cn=config, allow admin access to everything and defer to later decisions for everyone else. The “break” keyword induces the system to continue processing for the wildcard case. The restriction to users should mean that our LDAP database is a little harder to harvest for email addresses:

    olcAccess: {0}to *
  by dn.regex="^cn=admin,uid=[^,]+,ou=People,dc=acm,dc=jhu,dc=edu$" manage
  by dn.regex="^cn=[^,]+,uid=ldap,ou=Daemons,dc=acm,dc=jhu,dc=edu$" read
  by * none
  break

  • Allow anyone to check on our replication status (used by nagios):

    olcAccess: {1}to attrs=contextCSN,entryCSN,entryUUID by * read

  • Grant everyone the right to see the top-level of our chunk of the universal directory (specifically, that the entry and immediate children exist and what their objectTypes are):

    olcAccess: {2}
  to dn.exact="dc=acm,dc=jhu,dc=edu" attrs=entry,children,objectClass
   by * read

  • Allow users to change their login shell and GECOS information and allow everyone to read these fields:

    olcAccess: {3}
  to dn.subtree="ou=People,dc=acm,dc=jhu,dc=edu" attrs=loginShell,gecos
   by self write
   by * read

  • For the mail address, allow users to change it but only permit authenticated reads:

    olcAccess: {4}
  to dn.subtree="ou=People,dc=acm,dc=jhu,dc=edu" attrs=mail
   by self write
   by users read

  • Grant access to the fields that LDAP-as-NIS clients read to everyone for reading (since these clients are, in our setup, often anonymous):

    olcAccess: {5}
  to dn.subtree="ou=People,dc=acm,dc=jhu,dc=edu"
     attrs=entry,cn,gidNumber,uidNumber,homeDirectory,uid,objectClass
  by * read

  • Allow non-SASL auth to work, since we don’t otherwise use passwords in LDAP. Everyone’s userPassword field is boring: {SASL}...@ACM.JHU.EDU

    olcAccess: {6}
  to dn.subtree="ou=People,dc=acm,dc=jhu,dc=edu" attrs=userPassword
  by anonymous auth

  • Allow only the door server to search for users by card swipe (and Felica IDm), but allow nothing (not even the door server, for other purposes) to even see that they’re there:

    olcAccess: {7}
  to dn.subtree="ou=People,dc=acm,dc=jhu,dc=edu" attrs=jhuacmDoorCard,jhuacmFelicaIdm
  by peername.ip=10.161.159.219 search

  • Allow nobody (except the sysadmins because of the earlier rule) to see card swipe comments:

    olcAccess: {8}
  to dn.subtree="ou=People,dc=acm,dc=jhu,dc=edu" attrs=jhuacmDoorCardComment
  by * none

  • Grant access to fields that LDAP-as-NIS clients read for group information:

    olcAccess: {9}
  to dn.subtree="ou=Group,dc=acm,dc=jhu,dc=edu"
     attrs=entry,cn,gidNumber,memberUid,objectClass
  by * read

  • For objects representing service principals, grant access to the fields that LDAP-as-NIS clients read:

    olcAccess: {10}
  to dn.subtree="ou=Daemons,dc=acm,dc=jhu,dc=edu"
     attrs=entry,cn,gidNumber,uidNumber,homeDirectory,uid,objectClass,loginShell,gecos
  by * read

  • Allow read access by any authenticated entity:

    olcAccess: {11}to * by users read

Note

We used to have an IP-based catch-all, like this

olcAccess: {4}to * by users read by peername.regex=128\.220\.70\..+ read by * none

but we dropped that in favor of a more semantic approach ala above.

In the event that /admin access fails to work properly (for example, if cn=config got poked in a way that accidentally hosed that bit of configuration), the local root user on typhon likewise holds Real Ultimate Power(tm) via Unix-domain-socket LDAP (ldapi:///). Or, if things have truly hit the fan, you can resort to just modifying the LDIF files under /etc/ldap/slapd.d directly (but STOP SLAPD FIRST!)

TLS

We have certificates for ldap.acm.jhu.edu; sadly, none of our LDAP servers are actually called that! So until some common sense takes over the world, it may be necessary to add TLS_REQCERT allow to /etc/ldap/ldap.conf so that clients do not balk at the certificates if you’re going to refer to our LDAP servers by name, rather than by the ldap.acm.jhu.edu alias. That’s probably OK, as things requiring specific names should probably be using GSSAPI, but for example Nagios checks are not that smart and we want them watching over the SSL cert anyway!

That said, we should expect this command to work just fine:

ldapsearch -ZZ -x -H ldap://ldap.acm.jhu.edu -b dc=acm,dc=jhu,dc=edu

The certificates and keys and so on live in /etc/ldap/certs on all replicas; the olc database is configured appropriately:

dn: cn=config
olcTLSCACertificateFile: /etc/ldap/certs/ldap-201406-chain.pem
olcTLSCertificateFile: /etc/ldap/certs/ldap-201406.crt
olcTLSCertificateKeyFile: /etc/ldap/certs/ldap-201406.key
olcTLSVerifyClient: never

Replication

Oi! What a pain in the neck. I (nwf) read the following articles to help me get my head around it; these are listed in hopes that they’re useful to you, too, but really, I hope you never have to even think about it.

Changes to the database:

Load the syncprov module:

dn: cn=module{0},cn=config
olcModuleLoad: {1}syncprov

Create the syncprov stanza in the databases:

dn: olcOverlay={0}syncprov,olcDatabase={0}config,cn=config
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
olcSpCheckpoint: 20 60
olcSpSessionlog: 1000

dn: olcOverlay={0}syncprov,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
olcSpCheckpoint: 20 60
olcSpSessionlog: 1000

Create server identifiers. Unfortunately, these have to match how things are invoked on the command line, so we’re getting a little leakage of details that we probably shouldn’t be repeating here. It’s a little sad that we have to do this, as we never need to refer to the slaves, but seemingly…

dn: cn=config
olcServerID: 1 ldap://ldap1.acm.jhu.edu
olcServerID: 2 ldap://chicago.acm.jhu.edu:10389

Insert replication directives. The rid=N should match the master, as should the authcid="..." component.

dn: olcDatabase={0}config,cn=config
olcSyncrepl: {0}rid=1 provider=ldap://ldap1.acm.jhu.edu type=refreshAndPersist
  retry="60 30 300 +" searchbase="cn=config" bindmethod=sasl saslmech=gssapi
  realm=ACM.JHU.EDU authcid="ldap/typhon.acm.jhu.edu@ACM.JHU.EDU"

dn: olcDatabase={1}hdb,cn=config
olcSyncrepl: {0}rid=1 provider=ldap://ldap1.acm.jhu.edu type=refreshAndPersist
  retry="60 30 300 +" searchbase="dc=acm,dc=jhu,dc=edu" bindmethod=sasl
  saslmech=gssapi realm=ACM.JHU.EDU authcid="ldap/typhon.acm.jhu.edu@ACM.JHU.EDU"

Warning

Failure to renew Kerberos tickets correctly, or to configure Kerberos keytabs correctly as per above, will often result in a configuration that seems to work for a while, but will fail to replicate “eventually”, suspiciously near a key lifetime later. If GSSAPI auth fails or things go south otherwise (you forgot to make the jhuacmKerberosInstance DN for the replica, for example), the replica may clone only the public fields! Double check your work.

Happy LDAPing, and may your ldapsearches be ever fruitful. ~stump