Showing posts with label error 503. Show all posts
Showing posts with label error 503. Show all posts

Friday, October 2, 2015

Fedora Project's RHEL yum repo has been throwing errors since yesterday UPDATED

A few of my Red Hat servers run cron jobs to check for updates. starting yesterday (Thursday October 1st, 2015) at around 3PM I encountered 503 unavailable errors when attempting to contact a Fedora Project URL that hosts the metalink for the rhui-REGION-rhel-server-releases repository - a core RHEL repository for EC2.

Could not get metalink https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64 error was
14: HTTPS Error 503 - Service Unavailable

3 hours later or so, the URL began responding again, but the problems remained. `yum` now reports corrupted update announcements from the repo:

Update notice RHSA-2014:0679 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhui-REGION-rhel-server-releases repository.
Update notice RHSA-2014:1327 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHEA-2015:0372 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0335 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHEA-2015:0371 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0416 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0303 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0556 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0290 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0596 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0578 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0716 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1115 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1533 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1586 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1705 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.

I sent a tweet to Fedora to hopefully get some feedback. Because this wasn't a super critical issue I've been slacking on troubleshooting as well I will update here and/or provide a new post with more info.

UPDATE: I am increasingly convinced that this is an error with the repository and not something with my server. Check out the following command output:

Nothing marked as out of sync:
# yum distro-sync
Loaded plugins: amazon-id, rhui-lb
No packages marked for distribution synchronization

No problems listed by `package-cleanup`:
#package-cleanup --problems
Loaded plugins: amazon-id, rhui-lb
No Problems Found

`yum check` finds nothing:
# yum check
Not loading "rhnplugin" plugin, as it is disabled
Loading "amazon-id" plugin
Not loading "product-id" plugin, as it is disabled
Loading "rhui-lb" plugin
Not loading "subscription-manager" plugin, as it is disabled
Config time: 0.012
Yum version: 3.4.3
rpmdb time: 0.000
check all

The cache has been cleaned (repeatedly):
# yum clean all
Not loading "rhnplugin" plugin, as it is disabled
Loading "amazon-id" plugin
Not loading "product-id" plugin, as it is disabled
Loading "rhui-lb" plugin
Not loading "subscription-manager" plugin, as it is disabled
Config time: 0.021
Yum version: 3.4.3
Cleaning repos: epel rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-optional rhui-REGION-rhel-server-releases rhui-REGION-rhel-server-rh-common
Cleaning up everything

No orphans:
# package-cleanup --orphans
Not loading "rhnplugin" plugin, as it is disabled
Loading "amazon-id" plugin
Not loading "product-id" plugin, as it is disabled
Loading "rhui-lb" plugin
Not loading "subscription-manager" plugin, as it is disabled
Config time: 0.012
Setting up Package Sacks
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/supplementary/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/extras/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/supplementary/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rhscl/1/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/rhui-client-config/rhel/server/7/x86_64/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rhscl/1/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rhscl/1/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/extras/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/optional/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/optional/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/supplementary/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/optional/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/extras/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os
pkgsack time: 0.327
rpmdb time: 0.000
atomic-release-1.0-19.el7.art.noarch

By default, EC2 instances automatically repopulate mirrorlist URLs configured in /etc/yum.repos.d/*.repo files using the region in which the instance is hosted, like this:

mirrorlist=https://rhui2-cds01.REGION.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/$releasever/$basearch/os

I've manually updated the relevant .repo file with my region and upped the debugging level variables for yum-cron to try to narrow things down a bit. No answers yet ...

LATEST UPDATE (11-19): I believe I somewhat figured this out quite a while ago, but I just haven't had the time to update this post.

Amazon manages the licensing information for EC2 instances with operating systems that require it - like Windows and RHEL. So, the short answer is: Amazon broke it. I can't remember off-hand what the licensing agreement is in relation to this particular issue. I do know that I was still paying the exorbitant monthly rate for an RHEL-licensed instance. And I certainly received no notification that my RHEL license was expiring.

This was a very bad experience. The fact is, there are very few reasons why a non-enterprise scale user would ever use RHEL as opposed to CentOS. For Enterprise users that do require licensing, I would highly recommend looking into a Satellite-based updating solution. I'm not sure ATM what the logistics of doing such a thing using a platform like Amazon, but I am sure to be doing my homework on the subject shortly.

RAT Bastard

Earlier this week, several servers I maintain were targeted by automated attempts to upload a remote access trojan (RAT). The RAT is a simpl...