iDRAC email not sending - tests fail

 I was recently trying to configure iDRAC 9 on some servers to send email after discovering some unreliability in the inlet temperature of some servers located in someone else's server room.

Configuring the mail gateway in iDRAC involves setting its name, port, whether or not to use one of a couple encryption standards for the connection. and any required authentication. In another section of the UI, you can enter an email address and enable (State checkbox checked) and apply it. Then you can send a test email from that UI.

 My test emails were failing with the error:

RAC0225: Sending the test mail failed

 

At first I thought I might have the wrong SMTP gateway settings but double-checking confirmed that what I had specified (hostname and IP with no auth and no encryption) was working elsewhere.

 Then I thought it have been a firewall issue. I dug into pf rules on the relevant OpenBSD firewall and after not seeing anything obvious that would block that traffic I got to a spot where I was logging on the default block rule (block log all) and watching the log interface with tcpdump (tcpdump -n -e -ttt -i pflog0 src host myiDRACHost) and not seeing any blocked traffic. Maybe some other rule was blocking it... or maybe there was no traffic. While hunting for more info I came across this reply on a thread about the RAC0225 error above.https://www.dell.com/community/en/conversations/rack-servers/rac0225-sending-the-test-mail-failed-idrac/647f826cf4ccf8a8de123ad7?commentId=647f9939f4ccf8a8dec81250

 Maybe my iDRAC didn't have a DNS server set and therefore couldn't resolve my SMTP host. After digging into my iDRAC networking settings, that turned out to be the cause of the error for me. Once I set my DNS server IP in iDRAC, testing the alert emails resulted in a "Success!" message from iDRAC; however... I did not receive the test emails despite trying a couple addresses.

 Hmm.... off to the mail gateway to have a look in its logs. That turned up this error message:

status=bounced (host ASPMX.L.GOOGLE.COM[142.250.31.27] said: 550-5.7.1 [my_mail_gateway_IP] Messages missing a valid Message-ID header are not 550-5.7.1 accepted. For more information, go to 550-5.7.1  https://support.google.com/mail/?p=RfcMessageNonCompliant and review 550 5.7.1 RFC 5322 specifications. af79blahbe357-85cblahdeb7si19blah85a.553 - gsmtp (in reply to end of DATA command)) 

 Searching for that message revealed plenty of online chatter about the change in GMail policy. iDRAC was mentioned by a few people as giving them this same problem. Interestingly, my search results on DuckDuckGo included a response generated by Duck.ai. Expanding it revealed this:

Modify Postfix Settings (if applicable)

If you are using Postfix as your SMTP server, you can add the following line to your main.cf configuration file:

• always_add_missing_headers = yes

This setting instructs Postfix to add missing headers to incoming messages.

 After double-checking the postconf man page, I applied this setting, restarted postfix, sent another test message, and received the email!

ssh to old cisco switches and other old sshd implementations from newer ssh clients (version 8.8+)

I recently ran into a problem connecting to some cisco nexus switches that have an older sshd implementation that only works with ssh-rsa public keys and SHA1 checksums.

The initial symptom was being unable to login using the public key method with a  "too many authentication failures" error. After chasing that down and reducing the number of keys offered by my ssh agent from 4 to 2, the problem then became that it would reject both and move on to password auth. Suddenly I was seeing password prompts instead of the previously working ssh-rsa public key auth taking me straight into a shell on the switch.

Running ssh verbosely with ssh -v showed that my ssh-rsa key (only still around for use in these switches) was not being accepted. Instead the verbose output showed:

debug1: Offering public key: id_rsa RSA SHA256:ILXl4YsDBLAHBLAHBLAHBLAmPhz/D0Et1TBsClg agent
debug1: send_pubkey_test: no mutual signature algorithm

And after it looked for more public key files on disk to try, it got to:

debug1: Next authentication method: keyboard-interactive
(amos@123.456.789.012) Password: 

Some more digging led me to:

https://superuser.com/questions/1778874/openssh-v8-client-talking-to-openssh-v6-7p1-server-no-mutual-signature-algorit

And in turn:

https://www.openssh.com/txt/release-8.8

with this lovely bit:

Potentially-incompatible changes
================================

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken, and it is possible to create chosen-prefix
hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is
no need to replace ssh-rsa keys. OpenSSH has supported RFC8332
RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys
will automatically use the stronger algorithm where possible.

Incompatibility is more likely when connecting to older SSH
implementations that have not been upgraded or have not closely tracked
improvements in the SSH protocol. For these cases, it may be necessary
to selectively re-enable RSA/SHA1 to allow connection and/or user
authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms
options. For example, the following stanza in ~/.ssh/config will enable
RSA/SHA1 for host and user authentication for a single destination host:

    Host old-host
        HostkeyAlgorithms +ssh-rsa
	PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519).

And although I already had a .ssh/config file section for these switches with the `HostkeyAlgorithms +ssh-rsa` directive and a couple others, I did not have the `PubkeyAcceptedAlgorithms +ssh-rsa` directive. Adding it allowed my public key auth connections to my older cisco switches work again.

In the end, my working .ssh/config section for these switches looks something like:

 Host '123.456.789.*'
        HostkeyAlgorithms +ssh-rsa
        PubkeyAcceptedAlgorithms +ssh-rsa
        KexAlgorithms +diffie-hellman-group1-sha1
        Ciphers +aes128-cbc
        IdentityAgent ~/.1password/agent.sock

Get sshd to listen on multiple ports when systemd sockets are in use (affecting at least some recent Debian and Ubuntu containers)

I was going crazy trying to have sshd listen on multiple ports in a Debian Linux container under Proxmox Virtual Environment. This had been working at some point before on Debian Bookworm just by specifying multiple `Port` lines in /etc/ssh/sshd_config. But something changed at some point (either in Debian Bookworm updates or the upgrade to Debian Trixie) that changed how sshd is handled and caused it to appear to sometimes work and sometimes not.

I finally found this rude but helpful ServerFault answer:

https://serverfault.com/a/1142005/997178

It explains that sshd listen addresses and ports are now configured using systemd sockets. Setting them in sshd_config does nothing.

See /usr/share/doc/openssh-server/README.Debian.gz (use zcat) and pay special attention to the section near the end on systemd sockets.

Apparently this has been the default in Ubuntu for a while and recently became the pattern for Debian too.

Also see https://manpages.debian.org/stable/systemd/systemd.socket.5.en.html for info about sockets and the ListenStream option.

The final solution for me was to create `/etc/systemd/system/ssh.socket.d/listen.conf` containing:

[Socket]

#Clear ListenStream:

  ListenStream=

#Set new values. Multiple allowed:

  ListenStream=22

  ListenStream=2222

Growing Ubuntu LVM root filesystem inside a VM under Proxmox PVE

I wanted to grow the root filesystem of an Ubuntu 18.04 VM hosted in Proxmox PVE from 500GB to 2TB.

The VM is well backed up using Proxmox Backup Server and I have been through similar efforts with other VMs in other environments so I felt confident enough to proceed. 

I started by shutting down the VM and using the PVE GUI  (Hardware > Hard Disk > Disk Action > Resize) and specifying 2000 in the GB box. Then I started the VM, confirmed the change and shutdown again to take a snapshot before proceeding.

After booting the VM again I started to work through what I thought would be a relatively straightforward process of growing the partition, growing the physical volume, growing the logical volume, and then growing the filesystem.

It wasn't working as well as I expected. Searching online showed how to add a new partition of type `8e` (Linux LVM) to consume the free space, then creating a physical volume on it, then adding it to the volume group, then extending the root volume.

I thought this was a bit messy and thought there must be a way to just grow the existing partition without having to create a new one. Running `growpart /dev/sda 5` seemed to suggest that it was able to add a bunch of space to the partition but none of the pv, vg, and lv display, extending, or resizing commands seemed to suggest that the free space was available or that growth was possible.

In the end I perhaps jumped the gun by removing the swap LV thinking that it was blocking the growth of the root LV. Then I realized that the /dev/sda5 partition hosting the VG was an extended partition and that /dev/sda2 (the primary partition assigned to hold extended partitions) might need to be grown before /dev/sda5.

So what I ended up with was more or less:

 

swapoff -a

lvremove hostname-vg/swap_1 

growpart /dev/sda 2 

growpart /dev/sda 5

pvresize /dev/sda5 

lvextend hostname-vg/root /dev/sda5

resize2fs /dev/hostname-vg/root

 

But I'm not sure that I actually needed to remove the swap and was maybe just missing the `growpart /dev/sda 2` before the `growpart /dev/sda 5` and the rest to make it work. The next time I need to do this... try without removing swap first.