AWS Session Manager connect to EC2 instance

The most common way to manage a remote machine is SSH (Unix/Linux, Mac) or PowerShell/RDP (Windows), which requires the remote machine to open the corresponding access port and firewall, credentials or SSH Key. When selecting an EC2 instance on AWS console, we can click the "Connect" button, which provides three connection options:

1. EC2 Instance Connect: Requires EC2 to be configured with SSH Key, sshd is started, ssh inbound port allowed by Security Group, ec2-instance-connect installed(sudo yum install ec2-instance-connect)
2. Session Manager: This is what we are going to talk about next. sshd is not required(SSH key is not needed of cause). Security Group only requires outbound port 443. 
3. SSH client: Client SSH to EC2 instance, start sshd, allow inbound ssh port 22 by Security Group, use SSH Key or username and password in AMI, or configure to login with domain account after joining the domain.

AWS Session Manager provides access to EC2 instances through a browser or AWS CLI, and even machines or virtual machines in the local datacenter (requires advanced-instances tier support) , and no longer depends on SSH.

Session Manager Overview

Session Manager determines who can or cannot be accessed by the IAM access policy. It can be forwarded through the local port, the operation log in the session can be recorded as an audit, and can configure to send a message to Amazon EventBridge or SQS when session open or closed. The session log encrypted by a KMS key.

The basic condition for EC2 that supports Session Manager access is to run SSM Agent. SSM Agent has successively added support for various functions in its own version upgrade, such as KMS Key encryption, port forwarding and SSH sessions, log functions, etc. It is always better to install the latest version of SSM Agent.

SSM Agent will create a ssm-user user acts as a root user or super administrator(Windows). In order to start the SSM Agent on EC2, we should add the AmazonSSMManagedInstanceCore policy to the EC2 IAM Role, or add the required permissions by referring to Adding Session Manager permissions to an existing instance profile.

Session Manger console, go to https://console.aws.amazon.com/systems-manager/, click Start session, and select the EC2 instance with SSM Agent started from the list. Or on the EC2 instance console https://console.aws.amazon.com/ec2/, select an EC2 instance, click the Connect button, if the instance gets SSM Agent running, we are able to choose to use Session Manager to connect it.

AWS CLI: to use command ssm: aws ssm start-session --target i-0c0072d1212832d20 to connect the EC2 instance, we need to install Session Manager plugin for AWS CLI locally.

If we see Session Manger is enabled for an EC2 instance, that is because someone has configured the EC2 to install and start SSM Agent(or chosen AMI has built-in SSM Agent), and corresponding IAM role permission added. Let’s do it by ourselves from scratch on how to start an EC2 instance supports Session Manager connection.

Session Manager connects to EC2 instance

We'll take the image of Amazon ECS-Optimized Amazon Linux 2 AMI (2.0.20210121) as an example, please refer to Manually install SSM Agent on EC2 instances for Linux for details.

First, we need to create an IAM role to start EC2, and attach the AmazonSSMManagedInstanceCore policy to the IAM role. We name the IAM role as es2_role_with_ssm_agent. Start an EC2 instance and select above AMI image, then select or create a Security Group to allow outbound port 443 access.

Now, after the EC2 instance created, the button Connect of Session Manger becomes usable, click it to start Session Manager session on Web console. Log in user is  ssm-user, which can execute command with sudo.

After logging in, let's inspect ssm-agent process, current user, and check if sshd service started

Here are the necessary conditions for EC2 instances to be connected by Session Manager:

1. EC2 IAM role requires that the AmazonSSMManagedInstanceCore policy to be attached, or the corresponding permissions are manually added. This will create ssm-user with root permissions when use sudo.
2. Security Group requires EC2 to be able to connect to external port 443, such as 0.0.0.0:443. EC2 will connect to Session Manger servers(proxies) on port 443, no sshd service is required
3. EC2 must start ssm-agent process. From above screenshot, we see there are two processes /usr/bin/amazon-ssm-agentand /usr/bin/ssm-agent-worker listed. one ssm-session-worker process interact with one coming connection.

Note: If we don't include AmazonSSMManagedInstanceCore policy in EC2 IAM role permission,  we can start ssm-agent, but user ssm-user is not created. No user for Session Manager session, result in unable to connect to the EC2.

the AMI Amazon ECS-Optimized Amazon Linux 2 AMI (2.0.20210121) we chose has built-in ssm-agent, while it needs AmazonSSMManagedInstanceCore policy to create ssm-user user.

If selected AMI doesn't have build-in ssm-agent, let's take Amazon Linux 2 as an example, you need to add the following line to the userdata of EC2

sudo yum install -y https://s3.us-east-1.amazonaws.com/amazon-ssm-us-east-1/latest/linux_amd64/amazon-ssm-agent.rpm

Choose any region for this s3. Different Linux distributions have slightly different installation methods

ssm-agent would start automatically after installation

Other ways to connect to Session Manager

Earlier, we can directly find the EC2 instance, then connect to the Session Manager. We can also go to the Session Manager console to select the EC2 instance with ssm-agent started to initiate the connection. Here are several other ways to use Session Manager off Web console.

Start AWS CLI session locally

In addition to installing the AWS CLI locally, we need to install the Session Manager Plugin ( (Optional) Install the Session Manager plugin for the AWS CLI , find your own platform requirements). After that, just specify the EC2 instance ID instead of IP address to connect Session Manager.

$ aws ssm start-session --target i-0c0072d1212832d20
Starting session with SessionId: yanbin@example.com-0f3cabb8dd1bf0bb4
sh-4.2$ hostname
p-172-31-60-8.ec2.internal

This approach looks more professional. It looks like we are using SSH client. In fact, sshd service is not required on the EC2 host. If Session Manager Plugin is not installed, it will prompt below when running aws ssm start-session

SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-found

After Session Manager connected to EC2 instance, we can verify if sshd started or not, closing the SSH service won't impact Session Manager session.

$ telnet 10.255.57.211 22
Trying 10.255.57.211...

$ aws ssm start-session --target i-0c0072d1212832d20

Starting session with SessionId: yabqiu@gmail.com-06065c404df8b1cbe
sh-4.2$ sudo service sshd status
openssh-daemon is stopped
sh-4.2$ netstat -na|grep :22
sh-4.2$

Connect again with Session Manager

$ aws ssm start-session --target i-0c0072d1212832d20
Starting session with SessionId: yanbin@example.com-0f3cabb8dd1bf0bb4
sh-4.2$ hostname
p-172-31-60-8.ec2.internal

To further explore how Session Manager communicates with EC2 instance, check network connections on both local machine and EC2 instance.

So it is easy to get the following connection process

This can help us understand why Session Manager does not need SSH and open the corresponding port 22

Client SSH over session manager

Link: Step 8: (Optional) Enable SSH connections through Session Manager

Add below into the client configuration file ~/.ssh/config

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Now the EC2 instance needs to open the sshd service, but security group of inbound port 22 is still not needed.

Execute command on local

ssh -i ~/path/my-key-pair.pem ec2-user@i-0c0072d1212832d20
......
[ec2-user@ab-0aff39d3 ~]$

scp also works, use command scp -i /path/my-key-pair.pem test.txt ec2-user@i-0c0072d1212832d20

The only difference between this way and the above is that the ssm-agent on EC2 communicates with the EC2 native SSH service instead of directly interacting with the local shell.

Connection Diagram

Port forwarding service

Very similar to SSH Tunnel (ssh -A -L :56789:10.255.57.64:80), we can also leverage Session Manager for port forwarding, the command is as follows:

aws ssm start-session \
    --target instance-id \
    --document-name AWS-StartPortForwardingSession \
    --parameters '{"portNumber":["80"], "localPortNumber":["56789"]}'

Visit http://localhost:56789 to access port 80 of the remote EC2.

For more details of Session Manager documents, please refer to AWS System Manager documents .