Clearing out DHCP leases on (K)Ubuntu with dhcpd on OpenBSD.

 I have a lab of iMac computers at work. Recently I have been Installing Kubuntu on them and during the setup I decided to assign them specific hostnames and IPs from DHCP rather than just let them pick up the generic numbered names that my DHCP server was handing out by default.

My DHCP server is dhcpd on OpenBSD. I configured it to reduce the generic name pool and added static assignments for all the iMac MACs.  

Unfortunately, the systems already had leases. And releasing and renewing is not as easy in Kubuntu as in MacOS. Rebooting or disconnecting and reconnecting didn't pick up the new assignments and after a bit I realized that the server had cached the old leases and was renewing them rather than handing out the new static ones I had configured.

So for reference, here is what I did:

In OpenBSD, go through the /var/db/dhcpd.leases file and clean out the ones you don't want it to remember. Then `kill -HUP` the dhcpd process.

On the Kubuntu desktop, `sudo systemctl stop NetworkManager` , `sudo rm /var/lib/NetworkManager/*.lease`, and `sudo systemctl start NetworkManager`.

 Then the client got its new assigned address lease and all was well.

Installing Kubuntu on Late-2015 iMacs

Recently I have been installing and configuring Kubuntu on several older 27" 5K Retina iMacs (17,1 Late-2015) that are no longer supported with software updates or security patches by Apple. They are beautiful systems and have no issues other than being abandoned by Apple.

 These iMacs boot with UEFI and holding the Option key when powering on will offer a choice of boot devices, including the Kubuntu installer USB I flashed from the ISO download.

 Actually booting into the Kubuntu installer was a bit of a challenge though because there is a display power management issue with the AMD graphics in these iMacs that requires passing `amdgpu.dpm=0` to the kernel as an option when booting. If you don't do it, you might *eventually* see *something* to show up, but everything will be extremely slow.

Getting that kernel option in place can be a bit tricky if you can't catch the Grub bootloader in time. One way that helped me was to install Ventoy on the USB stick and then copy the Kubuntu ISO to it. Ventoy gave me a "Boot Kubuntu via Grub2" boot option. That gave me a chance to catch Grub with the ESC key and then `e` to edit the options and put `amdgpu.dpm=0` inline with the `quiet` and `splash` options that are were already there.

Proceeding with the install should cause the installer to install grub with that kernel option in place. But if you want to double check or find any painfully slow boot/display issues later on, you can always make sure that option is set in /etc/default/grub and run update-grub to put it back in place.

Ubuntu Gnome Launcher Missing Application Icons

 Recently, my non-snap version of Element (Desktop) on Ubuntu 26.04 started behaving badly by not displaying its icon properly in the Gnome "Dash" (see Gnome Terminology).

Some searching revealed Fixing Applications Icon Missing from the Launcher in Ubuntu and that led me to a fix by editing /usr/share/applications/element-desktop.desktop to switch the StartupWMClass to element rather than Element.

Unfortunately, some time later (likely after applying updates to the element-desktop package) the behaviour reverted to the broken icon.

I took a guess that copying that /usr/share/applications/element-desktop.desktop file to my ~/.local/share/applications/ folder and fixing it there might override the broken one being installed and maintained by the package and survive future upgrades to the package.

 My first hunch was correct. It overrides and works. Time will tell if it persists. In the meantime, Element issue #33472 exists for the problem caused by a mismatched StartupWMClass.

On a related note: Element issue #34895 points to Electron issue #53215 for the broken status icon in the Gnome "Top Bar".

No DHCP IP on eth0 with Ubuntu Server 26.04 (64-bit) on Raspberry Pi 4b

This week I was attempting to get up and running with Ubuntu Server 26.04 (64-bit) on a Pi 4b using Raspberry Pi Imager 1.8.5. I noticed this same problem both times I went through this process.

I used the option to edit and apply configuration settings in the Raspberry Pi Imager

General Tab:

• Set hostname

• Set username and password

• unchecked Configure wireless LAN

• set locale settings

Services:

• Enable SSH

• Allow Public Key Authentication Only (and pasted in my public key)

Then I flashed the microSD card, booted the Pi with it, watched a lot of stuff go by and then saw that it was trying and failing to reach the Ubuntu repos to update itself. The eth0 interface link light was on and maybe also flashing activity, but the system was unreachable on the network. Logging in on the console showed no assigned IP to eth0 even though /etc/netplan/50-cloud-init.yaml contained:

network:
  version: 2
  ethernets:
    eth0:
      optional: true
      dhcp4: true

In the end, all I had to do was run sudo netplan apply from the console and it picked up its eth0 IP and other info via DHCP.

Disconnect and power off USB peripheral from the Linux command line

I have a headless Proxmox PVE node that I sometimes need to mount USB storage to. When I'm done with it I can unmount the filesystem on it easily enough with the umount command but then I still have the drive and partitions still connected to the system. (In my case, that looks something like this below with my nvme drives and partitions showing up afterwards.)

root@pm:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                            8:0    0   1.9T  0 disk 
└─sda1                         8:1    0   1.9T  0 part

 

In a modern desktop environment on Linux you can find buttons or menu options to 'eject' storage like this but it wasn't obvious to me how to do it from the command line for a USB HDD enclosure. [edit: so the `eject` manual page suggests that running `eject /dev/sda` might have also worked. I'll try it next time!]

So I did a Kagi search and found a solution on Reddit that worked.

From https://www.reddit.com/r/linuxquestions/comments/1dq4ha3/comment/lao9dnq/

Echoing the usb port number to /sys/bus/usb/drivers/usb/unbind works for regular usb devices, like external drives or hubs. It removes them from from the bus and stops them consuming power, just as if they were physically unplugged. You need both the bus number and port, which you can get via lsusb -t. For example if the device is on bus 01 port 3, you would use

echo 1-3 | sudo tee /sys/bus/usb/drivers/usb/unbind

Echoing the same to /sys/bus/usb/drivers/usb/bind plugs it back in.

 

Upgrade self-hosted GitLab instance on Ubuntu after falling too far behind

I recently ran into a signing key issue on an Ubuntu system running gitlab-ee from the official GitLab repos but didn't notice for a while so my normal automatic upgrades hadn't been working to keep it updated. This may have also been affecting other package upgrades... not sure yet.

Anyway, the first thing to solve was getting the latest repo key installed. Downloading and running the script mentioned in step 2 of https://docs.gitlab.com/install/package/ubuntu/?tab=Enterprise+Edition did the trick for the key. 

Afterwards, I was faced with the issue of being multiple Minor revisions behind. This manifested as this error:

Stop optical discs from automatically mounting in Ubuntu 25.10

 I was recently using MakeMKV to rip some old DVDs to watch using Jellyfin/Plex and ran into an issue where the drive seemed to be in contention between GNOME and MakeMKV.

Manually unmounting the DVD in the GNOME dock and restarting MakeMKV seemed to allow it to continue so I had a quick look for how to change that desktop normally useful behaviour while I was working on this task.

 I tried installing dconf-editor but trying to change the org/gnome/desktop/automount value to false gave me a cryptic error about something not being available for editing. Same with other values in that same section. It may have been because I was trying to run it as root rather than my current desktop user or something like that, but since the advice to use that tool was from a decade ago and I'm running Ubuntu 25.10, I had another look around.

I found the gsettings command line utility.

Running gsettings set org.gnome.desktop.media-handling automount false as my logged in user took effect immediately without logging out and in again and I didn't see the MakeMKV problems again.

Afterwards, I ran gsettings set org.gnome.desktop.media-handling automount true to revert to the normally useful default automount behaviour.

 

Predictable Network Names

It turns out that systemd's Predictable Network Interface Names are a bit of a misnomer.

The original change from eth0, eth1, etc. to the new v197 scheme described in the link above caught me out at one point way back during an Ubuntu Linux LTS upgrade [maybe from 14.04 to 16.04? I can't recall precisely] and left my system offline until I could get to the console and reconfigure the static IP network configuration from the old eth0 to the new interface name. I just switched the config over and continued to use the ens-prefixed name that systemd/udev picked.

More recently, a Proxmox Virtual Environment upgrade caught me out and I researched more. 

I understood that systemd/udev attempts to assign predictable names by using a versioned list of attributes to assemble the interface names in a predictable way. What I discovered was that unfortunately, not all attributes used in a given version are available from all kernels and so even if a system upgrade sticks to the same version of systemd/udev, a new or different kernel or driver version may add or drop support for reporting some of the attributes. This can result in a name being different after a kernel change.

Having found a good solution to (un)Predicatable Network Names that involves assigning a custom interface name based on MAC address, I implemented it on the various PVE hosts I was managing so that a future upgrade that affected the network interface naming would not cause havok.

I'm not on my 3rd pre-emptive fix on yet another system and had to hunt for the systemd.link docs again after not finding the recipe in this blog. So here it is:

Create an ordered collection of .link files, one per interface. Similar to other files processed in numerical order, by convention start with a two digit numeric prefix (in my case "10-") followed by the name you have chosen for the interface (in my case "en10g0" representing my ethernet 10Gb interface #0) and tack a .link on the end. Put it in the `/etc/systemd/network` directory.

e.g. `/etc/systemd/network/10-en10g0.link`

The contents of the file in my case would be something akin to:

[Match]

MACAddress=00:11:22:33:44:55

 

 [Link]

Name=en10g0 

If you get all your interface names configured similarly, then you can use those names everywhere else they get hard coded (such as in the `/etc/network/interfaces` file for Linux bridge devices, etc.) and then when you have caught and replaced them all (and have root access to the console just in case you missed something), reboot. If all is well, the network should come back up OK and then you should not have to worry about your interface names changing unless you change any hardware or MAC addresses. Personally, I feel that after messing with networking hardware is a much better time for networking to break because I'm alert and expecting it.

iDRAC email not sending - tests fail

 I was recently trying to configure iDRAC 9 on some servers to send email after discovering some unreliability in the inlet temperature of some servers located in someone else's server room.

Configuring the mail gateway in iDRAC involves setting its name, port, whether or not to use one of a couple encryption standards for the connection. and any required authentication. In another section of the UI, you can enter an email address and enable (State checkbox checked) and apply it. Then you can send a test email from that UI.

 My test emails were failing with the error:

RAC0225: Sending the test mail failed

 

At first I thought I might have the wrong SMTP gateway settings but double-checking confirmed that what I had specified (hostname and IP with no auth and no encryption) was working elsewhere.

 Then I thought it have been a firewall issue. I dug into pf rules on the relevant OpenBSD firewall and after not seeing anything obvious that would block that traffic I got to a spot where I was logging on the default block rule (block log all) and watching the log interface with tcpdump (tcpdump -n -e -ttt -i pflog0 src host myiDRACHost) and not seeing any blocked traffic. Maybe some other rule was blocking it... or maybe there was no traffic. While hunting for more info I came across this reply on a thread about the RAC0225 error above.https://www.dell.com/community/en/conversations/rack-servers/rac0225-sending-the-test-mail-failed-idrac/647f826cf4ccf8a8de123ad7?commentId=647f9939f4ccf8a8dec81250

 Maybe my iDRAC didn't have a DNS server set and therefore couldn't resolve my SMTP host. After digging into my iDRAC networking settings, that turned out to be the cause of the error for me. Once I set my DNS server IP in iDRAC, testing the alert emails resulted in a "Success!" message from iDRAC; however... I did not receive the test emails despite trying a couple addresses.

 Hmm.... off to the mail gateway to have a look in its logs. That turned up this error message:

status=bounced (host ASPMX.L.GOOGLE.COM[142.250.31.27] said: 550-5.7.1 [my_mail_gateway_IP] Messages missing a valid Message-ID header are not 550-5.7.1 accepted. For more information, go to 550-5.7.1  https://support.google.com/mail/?p=RfcMessageNonCompliant and review 550 5.7.1 RFC 5322 specifications. af79blahbe357-85cblahdeb7si19blah85a.553 - gsmtp (in reply to end of DATA command)) 

 Searching for that message revealed plenty of online chatter about the change in GMail policy. iDRAC was mentioned by a few people as giving them this same problem. Interestingly, my search results on DuckDuckGo included a response generated by Duck.ai. Expanding it revealed this:

Modify Postfix Settings (if applicable)

If you are using Postfix as your SMTP server, you can add the following line to your main.cf configuration file:

• always_add_missing_headers = yes

This setting instructs Postfix to add missing headers to incoming messages.

 After double-checking the postconf man page, I applied this setting, restarted postfix, sent another test message, and received the email!

ssh to old cisco switches and other old sshd implementations from newer ssh clients (version 8.8+)

I recently ran into a problem connecting to some cisco nexus switches that have an older sshd implementation that only works with ssh-rsa public keys and SHA1 checksums.

The initial symptom was being unable to login using the public key method with a  "too many authentication failures" error. After chasing that down and reducing the number of keys offered by my ssh agent from 4 to 2, the problem then became that it would reject both and move on to password auth. Suddenly I was seeing password prompts instead of the previously working ssh-rsa public key auth taking me straight into a shell on the switch.

Running ssh verbosely with ssh -v showed that my ssh-rsa key (only still around for use in these switches) was not being accepted. Instead the verbose output showed:

debug1: Offering public key: id_rsa RSA SHA256:ILXl4YsDBLAHBLAHBLAHBLAmPhz/D0Et1TBsClg agent
debug1: send_pubkey_test: no mutual signature algorithm

And after it looked for more public key files on disk to try, it got to:

debug1: Next authentication method: keyboard-interactive
(amos@123.456.789.012) Password: 

Some more digging led me to:

https://superuser.com/questions/1778874/openssh-v8-client-talking-to-openssh-v6-7p1-server-no-mutual-signature-algorit

And in turn:

https://www.openssh.com/txt/release-8.8

with this lovely bit:

Potentially-incompatible changes
================================

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken, and it is possible to create chosen-prefix
hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is
no need to replace ssh-rsa keys. OpenSSH has supported RFC8332
RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys
will automatically use the stronger algorithm where possible.

Incompatibility is more likely when connecting to older SSH
implementations that have not been upgraded or have not closely tracked
improvements in the SSH protocol. For these cases, it may be necessary
to selectively re-enable RSA/SHA1 to allow connection and/or user
authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms
options. For example, the following stanza in ~/.ssh/config will enable
RSA/SHA1 for host and user authentication for a single destination host:

    Host old-host
        HostkeyAlgorithms +ssh-rsa
	PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519).

And although I already had a .ssh/config file section for these switches with the `HostkeyAlgorithms +ssh-rsa` directive and a couple others, I did not have the `PubkeyAcceptedAlgorithms +ssh-rsa` directive. Adding it allowed my public key auth connections to my older cisco switches work again.

In the end, my working .ssh/config section for these switches looks something like:

 Host '123.456.789.*'
        HostkeyAlgorithms +ssh-rsa
        PubkeyAcceptedAlgorithms +ssh-rsa
        KexAlgorithms +diffie-hellman-group1-sha1
        Ciphers +aes128-cbc
        IdentityAgent ~/.1password/agent.sock

Get sshd to listen on multiple ports when systemd sockets are in use (affecting at least some recent Debian and Ubuntu containers)

I was going crazy trying to have sshd listen on multiple ports in a Debian Linux container under Proxmox Virtual Environment. This had been working at some point before on Debian Bookworm just by specifying multiple `Port` lines in /etc/ssh/sshd_config. But something changed at some point (either in Debian Bookworm updates or the upgrade to Debian Trixie) that changed how sshd is handled and caused it to appear to sometimes work and sometimes not.

I finally found this rude but helpful ServerFault answer:

https://serverfault.com/a/1142005/997178

It explains that sshd listen addresses and ports are now configured using systemd sockets. Setting them in sshd_config does nothing.

See /usr/share/doc/openssh-server/README.Debian.gz (use zcat) and pay special attention to the section near the end on systemd sockets.

Apparently this has been the default in Ubuntu for a while and recently became the pattern for Debian too.

Also see https://manpages.debian.org/stable/systemd/systemd.socket.5.en.html for info about sockets and the ListenStream option.

The final solution for me was to create `/etc/systemd/system/ssh.socket.d/listen.conf` containing:

[Socket]

#Clear ListenStream:

  ListenStream=

#Set new values. Multiple allowed:

  ListenStream=22

  ListenStream=2222