Configuring TLS in Elasticsearch is Insane

Of all the projects I've configured TLS on, Elasticsearch is by far the most obtuse and convoluted. I presume this is because this feature was added after the fact as part of the basic/open/free tier from behind the paywall.  But then was it this ridiculous as a paid feature? No idea.

It should be simple. For example, Kafka has you set the listener types and associated key/trust stores. Elastic (using 7.10) requires you to enable the feature from xpack, enable it at transport and/or http levels, add mode, client required, and then key/truststores. All of their examples use p12 (which makes sense given we're supposed to be moving away from jks, but I digress), but then it requires using the "elasticsearch" keystore, a flat file on disk to protect sensitive items like store passwords. If someone has physical access to the box, it makes no difference.

I could not get a "passwordless" p12 to work like they use in their examples. I presume this is user error, but p12 does still encrypt a key even if no passphrase is provided. The ssl error messages weren't helpful enough to diagnose, so I added passwords to the elastic keystore.

In summary, for elastic:

#turn it on
xpack.security.enabled: true
#turn on internode commms tls
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /path/to/p12
xpack.security.transport.ssl.truststore.path: /path/to/p12  

#turn on rest endpoint tls
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required
xpack.security.http.ssl.keystore.path: /path/to/p12
xpack.security.http.ssl.truststore.path: /path/to/p12  

#then you need to add passwords to the elastic keystore
#This has to be done on every node
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

Kibana is a little more straightforward:

#turn it on
server.ssl.enabled: true
#certificate the server presents through/to the browser
server.ssl.certificate: /path/to/cert/pem
server.ssl.key: /path/to/key/pem

#credentials kibana presents to elastic
elasticsearch.ssl.certificate: /path/to/cert/pem
elasticsearch .ssl.key: /path/to/key/pem
elasticsearch.ssl.certificateAuthorities: ["/path/to/ca/pem"]

One exception. I was hitting a "empty cert chain" issue when kibana hit elastic. The above should be sufficient...

BUT WAIT! There is one random setting you need:

elasticsearch.ssl.alwaysPresentCertificate: true

Why is this a setting? In what universe would I set up TLS certs and then not want to present one? Even if elastic was configured to optional for certs, why wouldn't kibana present one if it was set? This was an obscure forum post find.

And filebeat:

output.elasticsearch:
  protocol: "https"
  ssl.certificate_authorities:
    - /path/to/ca/pem
  ssl.certificate: /path/to/crt/pem
  ssl.key: /path/to/key/pem
  # also these if you have basic auth setup post cert check
  username: username
  password: password

The elastic documentation on this is fairly good, but somehow still lacking. I'm not sure what I would have them add except for more examples and what types each option allows.