It turned out to be an expired root password. The user "cabernocht" on the VMWare community forums figured it out.
I used SSH to log in to the VCSA system, get to the shell, and run chage --list root to check. Then I reset the root password with passwd and disabled password expiry and cleared password history with chage -M -1 root; echo "" > /etc/security/opasswd
After the fix, I tried the update again. The web GUI still disconnected during the "running pre-scripts" phase, but when I connected with SSH I was able to verify the log with less /var/log/vmware/applmgmt/software-packages.log and see that it had been applied. It had tried to output that the update had succeeded and that I should reboot but it had turned the web GUI off during the update so I'm not sure how that was supposed to work. Anyway, I ran reboot at the shell and after a few minutes, I was able to connect to the web GUI at port 5480 again and it reported the new version.
See https://communities.vmware.com/message/2815256#2815256
See https://communities.vmware.com/message/2815256#2815256