Problem Statement:

You are attempting to ssh into a Linux VM and are unsuccessful. We have seen customers modify key sshd configuration files and upon restarting sshd or rebooting the VM connectivity is lost.

symptoms/errors experienced

The serial or console log may exhibt errors such as this:

SuSE
/var/empty must be owned by root and not group or world-writable.
startproc: exit status of parent of /usr/sbin/sshd: 255
Failed

CentOS
Starting sshd: /var/empty/sshd must be owned by root and not group or world-writable.
[FAILED]

Resolution options:

There are a few methods to change the permissions on a Linux file when you don’t have ssh access.

1 – Portal – Run a Linux Custom Script into the VM which issues the Linux commands chmod on the file that you need to change via portal.azure.com

2 – Manually – Delete the VM, keep the disks, mount OS disk to another temporary VM and update the files on the temporary VM. Recreate the VM from disk

3 – Serial Console – Use the Serial Console feature if available for the region your VM is located in

4 – CLI – The command Line Interface can also be used to run BASH commands into your VM

1. Portal

This option relies on theWAagent process being up. From the serial log or portal check if the agent looks to be up.

Navigate to Properties of the VM in the portal to see if the WAagent is ok

Navigate to portal.azure.com and locate you VM’s settings. Go to “Extensions” -> “Add” select “Custom Script For Linux“

Select the “Create” button

Populate the “Script File” field with the attached script (download update_perms.txt to your local PC/MAC and rename to update_perms.sh)

Note you might have to update the script to reflect your OS – this attached script runs on Red Hat variants.

Notifications will update when the Script has completed – The STATUS should display Success

In the guest VM the example script update_perms.sh , changes the permissions on the file /var/empty/sshd from 777 to 755 and sets the owner and group to root:root

Wait for script to run. The script will take a few minutes. The Linux Agent receives the request and hands off to the correct extension.

If you are able to ssh back into your VM and are curious to see what happened during executing of the update_params.sh, look in the direcory /var/log/azure/Microsoft.OSTCExtensions.CustomScriptForLinux/<version> and check the file extension.log

2. Manually attach your VM’s disks to a temporary VM

Delete the VM, keep the disks, mount OS disk to another temporary VM and update the files (permissions and ownership) on the temporary VM. Recreate the VM from disk. Instructions on this method can be found here

3. Serial Console

This feature allows you to access the VM via a console as if you were “on-prem” or “bare metal” permitting you to modify files without using Custom Script and without having to delete the VM. Boot diagnostics need to be configured as a prerequisite.

4. CLI

The Command Line Interface (CLI) can be used to inject commands into a VM. There are 2 versions of CLI – latest documentation is located here

In this example, using the newer CLI v2 version we are resetting the /var/empty/sshd permissions and ownership using the customScript extension via CLI. The sshd service is also restarted.

You will need to replace yourvm with the name of Virtual Machine and yourrg with your Resource Group

vmname=yourvm;rg=yourrg;timestamp=`date +%d%Y%H%M%S`;az vm extension set –resource-group $rg –vm-name $vmname –name customScript –publisher Microsoft.Azure.Extensions –settings “{‘commandToExecute’: ‘bash -c \’chmod 755 /var/empty/sshd;chown root:root /var/empty/sshd;systemctl start sshd;ps -eaf | grep sshd\”,’timestamp’: “$((timestamp))”}”