Showing posts with label DNS. Show all posts
Showing posts with label DNS. Show all posts

Friday, January 8, 2016

Setting a hostname for your Amazon AWS EC2 server running RHEL or CentOS 7

So it turns out that setting your AWS EC2 server's hostname to be persistent across reboots is a surprising pain in the ass, at least with my usual OS of choice - RedHat/CentOS Linux.

If you're like me, setting a hostname is the sort of trivial non-task that at this point you really feel like you dont need to RTFM to figure out. You know about `hostnamectl set-hostname`. You've tried `nmcli general hostname`. You've manually set /etc/hostname. None of its persists past a reboot. Which can make life very difficult for those planning to use EC2 for email or dozens of other tasks.

Here's how to do it the right way, the first time. I'll also describe some circumstances that setting your own hostname will break things, and why its such a hassle to get this done in AWS in the first place.

Amazon relies on cloud-init to manage a variety of initialization tasks for its cloud servers; cloud-init was originally built to support Ubuntu images, but it is now used for a variety of different Amazon distros, including RHEL, CentOS and "Amazon linux". cloud-init is manged through a series of configuration files and modules; you can use them to add SSH keys, setup chef & puppet recipes, install SSL certificates, and all sorts of stuff. Think of it as a very fancy kickstart script.

By default, Amazon resets your server's hostname to the Public DNS entry for the IP address assigned to your server. These default hosts look something like this: for an IP address 111.222.333.444. If you have an Elastic IP Address, this hostname can be viewed through your EC2 Console by navigating to Network & Security -> Elastic IPs. The hostname is viewable in the "Public DNS" column. Because of this behavior, all of the default methods for assigning a hostname to your server are over-ridden on reboot. There is no way to change the hostname through the EC2 Console after your server has been built.

Here's the part of the walk through where I describe some circumstances where messing with your hostname can break stuff. If you have not assigned at least one Elastic IP Address (EIP) to your server, I strongly advise against messing with your server's hostname. Without an EIP, Amazon changes your server's public IP, private IP and hostname to whatever is available at the moment in your region. I haven't tried it, but I strongly suspect that making the changes in this walkthrough without an EIP will either just not work or will break something. There may be circumstances where you would want to accomplish this; hacks probably exist but this walkthrough ain't it.

Here's what to do:

Update the /etc/hostname file with your new hostname:
    [centos@... ~]$ sudo vi /etc/hostname
Initially, this file will contain the hostname assigned by Amazon. Delete this value and replace it with your preferred hostname. With vi, you must enter "INSERT MODE" to make changes to a document by pressing the i key.
NOTE: the official Amazon walkthrough tells you to add your hostname like this: HOSTNAME=persistent_host_name - that is incorrect. The correct way is to just put your hostname in there; if you want your hostname to be than the contents of /etc/hostname should be The official walkthrough also tells readers to use vim using the syntax #vim <filename>. Although installed by default with RHEL 7 & CentOS 7, vim has to be launched using #vi <filename>. 
Save and exit the vi editor. After you've made you're changes, press ESCAPE to exit INSERT MODE, then press SHIFT and : [colon] simultaneously to issue a command to the vi editor. Type wq, and then press Enter to save changes and exit back to the command prompt.

Update the /etc/hosts file with the new hostname.
    [centos@... ~]$ sudo vi /etc/hosts
Change the entry beginning with to read as follows: localhost.localdomain localhost
Save and exit the vi editor.

Update the /etc/sysconfig/network file.
    [centos@... ~]$ sudo vi /etc/sysconfig/network
Update the /etc/sysconfig/network file with the following values:
Save and exit the vi editor.
Change your server's primary cloud-init configuration file
    [centos@... ~]$ sudo vi /etc/cloud/cloud.cfg
Add the following string at the bottom of the file to ensure that the hostname change stays after a reboot.
    preserve_hostname: true
NOTE: At the bottom of /etc/cloud/cloud.cfg, you may find a line that appears to be commented out, like this: # vim:syntax=yaml - the preserve_hostname line must go at the very bottom of the file, even beneath the commented out line, or else it won't work.
Save and exit the vi editor.
Run the following command to reboot the instance to pick up the new hostname:
    [centos@... ~]$ sudo reboot 

After you reboot your server, execute the hostname command to check that your changes have stayed put.
    [centos@... ~]$ hostname
The command should return the new hostname:
    [centos@... ~]$ hostname

And that's about it, sports fans. I ripped off most of this from an Amazon KB article on the topic, with a few updates where the KB had some mistakes. This has been an issue with AWS for a while, and there appears to be a lot of confusion on the internet on how to get this accomplished, so I hope that by making this available more people will be able to get this resolved without wasting time.

Sunday, February 8, 2015


Hey all - I have been able to reacquire one of my domains that were so viciously stolen from me by domain squatters some time ago. So, you can now visit the site using Keep in mind that all links to will continue to work.

Saturday, October 4, 2014

Amazon EC2 Connectivity Failures - 10/4/2014

I have seen indications of periodic connectivity issues impacting Amazon's EC2 Cloud Computing architecture. Personally, I have encountered issues with connecting to Amazon's Yum repository hosts from EC2 instances.

Amazon has published Outage notifications of brief connectivity and DNS failures impacting US-EAST-1 Availability zone between October 2nd and October 4th. However, my EC2 instances are within the US-WEST-2 Availability zone and I am experiencing issues today, October 4th 2014 at approximately 11:30 AM EST.

For example:

# yum provides seinfo
Loaded plugins: amazon-id, rhui-lb

epel/x86_64/filelists_db                                        | 4.7 MB  00:00:01
rhui-REGION-rhel-server-optional/7Server/x86_64/filelists_db    | 3.2 MB  00:00:00 [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror. [Errno 14] HTTPS Error 404 - Not Found

Then, 5 minutes later, with absolutely no changes to my server's network or yum configuration:

# host has address

# yum provides seinfo
Loaded plugins: amazon-id, rhui-lb
setools-console-3.3.7-46.el7.x86_64 : Policy analysis command-line tools for SELinux
Repo        : rhui-REGION-rhel-server-releases
Matched from:
Filename    : /usr/bin/seinfo

I find this extremely frustrating. With my small presence on EC2, I have no ability to troubleshoot what is causing these issues. However, I can confirm that there *are* issues as of today, that Amazon has been aware of connectivity and DNS failures for at least two days, and that Amazon is currently claiming that there are no issues.

This is quickly becoming the industry-standard mode of behavior for Cloud computing providers: wild-eyed, outlandish promises of perfect availability followed by regular connectivity failures that are haphazardly brushed under the rug.

Customers are owed transparency. I remain convinced that the only way to accomplish reliability is by "doing it yourself" and colocating servers in multiple datacenters, implementing and managing redundancy directly. The issue is too important to trust to hosting providers who have consistently demonstrated dishonesty.

See for yourself the almost invisible notice Amazon has posted to customers on their Service Health Dashboard:

Amazon EC2 Buries Connectivity Failure Notifications
Downtime? What Downtime?

NSA Leak Bust Points to State Surveillance Deal with Printing Firms

Earlier this week a young government contractor named Reality Winner was accused by police of leaking an internal NSA document to news outle...