Skip to main content

Posts

Showing posts with the label EC2

EC2 swap device management & fixing "swapoff failed: Cannot allocate memory"

 One of the sillier things I've done as an AWS/linux admin is provision an EBS disk as swap to an EC2 instance. I kept getting max allocate errors for a script I needed to run to execute a series of database queries. Reprovisioning to a new EC2 instance class with more RAM wasn't feasible at the time for some long-forgotten reason.  I would never do this if I owned the disks - provisioning swap to SSD will greatly reduce the lifetime of the disk, among many reasons why this is less than ideal. But Amazon has plenty of money. I figured I could cheaply provision an EBS volume & buy myself enough swap to complete the query. Then, in some point in the future, I could create a more beautimous solution. Well, if you're a sysadmin you know how this story ends. I moved onto other fires/projects, quickly forgot about the swap situation, and here I am years later, deprovisioning the server, in all its swappy glory. This wouldn't warrant a blog post, except for the fact that I

Recovering network access to EC2 instances

So you've screwed something up. You made a typo in your sshd_config file. You added a firewall rule, or a route, or some other thing, and lost your network access to your EC2 instance. And of course whatever you broke, you broke permanently - you wrote your firewall rules directly to /etc/sysconfig/iptables, you made your goofy change to /etc/sysconfig/network-scripts/whatever-interface; so rebooting won't make a damn bit of difference. You read the warnings, you know you shouldn't have. But you did anyway. Oh, and you don't have any backups. Or you have backups from three months ago. Restoring from your crappy backups would mean hours to days of non-stop work and consistent downtime. Or Amazon or whatever other company you're using for backups actually broke your backups/lost your backups/never actually provided you with the backups you paid for. Don't panic. You've got this . You remember that Amazon has some sort of Java-based something or other. Its

Setting a hostname for your Amazon AWS EC2 server running RHEL or CentOS 7

So it turns out that setting your AWS EC2 server's hostname to be persistent across reboots is a surprising pain in the ass, at least with my usual OS of choice - RedHat/CentOS Linux. If you're like me, setting a hostname is the sort of trivial non-task that at this point you really feel like you dont need to RTFM  to figure out. You know about `hostnamectl set-hostname`. You've tried `nmcli general hostname`. You've manually set /etc/hostname. None of its persists past a reboot. Which can make life very difficult for those planning to use EC2 for email or dozens of other tasks. Here's how to do it the right way, the first time. I'll also describe some circumstances that setting your own hostname will break things, and why its such a hassle to get this done in AWS in the first place. Amazon relies on cloud-init to manage a variety of initialization tasks for its cloud servers; cloud-init was originally built to support Ubuntu images , but it is now used fo

EC2 IP aliasing script is now ready for use

About a month and a half ago I grew so frustrated by the boneheaded way that Amazon EC2 handles IP aliasing that I wrote a pretty lengthy post about the problems entailed and included a small program that would fix those problems . Amazon provides some pretty productive documentation for some types of users. There is help available for you if you are any one of the following:      - You are willing to pay for a new ENI to support a second IP address      - You are multihoming / load balancing      - You want to use "Amazon Linux" and install their ec2-net-utils But, if you want to just add a second IP address to a pre-existing Linux server, you are pretty much screwed. Well, you were screwed. Now you can install my program - aliaser - as a service and it will route additional IP addresses for you without the need for an extra ENI. I've uploaded aliaser to Github   - it includes a shell script and a .service file, as well as some very easy-to-follow instruction

An IRS tax refund phishing scam illustrates the widespread failure of hosting and antivirus providers' security measures

Scams focused on stealing tax refunds remain highly profitable, despite the fact that they are well known and understood by security professionals and the general public , and have been for years. A variety of distribution methods are used, with the common threads being the use of IRS logos and bureaucratic-sounding language to convince users to click a link, download and execute a file and/or send personally identifying information like a Social Security number. A recent example of one such a scam that I came across is a damning illustration of the failure of online service providers to protect users from obvious and simple malware distribution methods. In the example I wish to discuss today, the distribution method was a spammed email that on a small ISP's installation of SpamAssassin (note: I am not an admin or employee of this system; I'm a customer) received an X-Spam-Status score of 5.3 after being flagged with the following variables: X-Spam-Status: No, score=5.3 re

Assigning multiple IP addresses to a single Amazon EC2 instance on a single ENI

UPDATE March 1st, 2017 : I'm glad to see that people are finding this helpful, and thanks to everyone that has contacted me here or via email. Just to be clear, though, the script on GitHub works much better than what I describe here in this post. The idea for this post was to describe the basics of how to get IP aliasing working in EC2 w/out using Amazon's weirdo linux distro, and I wrote it about a while before I posted the script to GitHub . If you want functional code with step-by-step instructions, goto the aliaser GitHub repo . I just don't have the time to rewrite the post each time I (or someone else) has an update for the script. Also, if you have feature requests or feedback, it will be easier for me to get back to you on GitHub than here ... especially if you have something specific you want added or that doesn't work. Also, just FYI, I added a systemd .service file to the script in the aliaser GitHub repo a year ago. IIRC its LFB compatible so should wo

Amazon EC2 Connectivity Failures - 10/4/2014

I have seen indications of periodic connectivity issues impacting Amazon's EC2 Cloud Computing architecture. Personally, I have encountered issues with connecting to Amazon's Yum repository hosts from EC2 instances. Amazon has published Outage notifications of brief connectivity and DNS failures impacting US-EAST-1 Availability zone between October 2nd and October 4th. However, my EC2 instances are within the US-WEST-2 Availability zone and I am experiencing issues today, October 4th 2014 at approximately 11:30 AM EST. For example: # yum provides seinfo Loaded plugins: amazon-id, rhui-lb epel/x86_64/filelists_db         | 4.7 MB  00:00:01 rhui-REGION-rhel-server-optional/7Server/x86_64/filelists_db    | 3.2 MB  00:00:00 https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x86_64/os/repodata/e5ee2c196ee6525998525a2bf74bb40608dce199-filelists.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found T