Showing posts with label error. Show all posts
Showing posts with label error. Show all posts

Friday, October 2, 2015

Fedora Project's RHEL yum repo has been throwing errors since yesterday UPDATED

A few of my Red Hat servers run cron jobs to check for updates. starting yesterday (Thursday October 1st, 2015) at around 3PM I encountered 503 unavailable errors when attempting to contact a Fedora Project URL that hosts the metalink for the rhui-REGION-rhel-server-releases repository - a core RHEL repository for EC2.

Could not get metalink https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64 error was
14: HTTPS Error 503 - Service Unavailable

3 hours later or so, the URL began responding again, but the problems remained. `yum` now reports corrupted update announcements from the repo:

Update notice RHSA-2014:0679 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhui-REGION-rhel-server-releases repository.
Update notice RHSA-2014:1327 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHEA-2015:0372 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0335 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHEA-2015:0371 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0416 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0303 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0556 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0290 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0596 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0578 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0716 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1115 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1533 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1586 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1705 (from rhui-REGION-rhel-server-releases) is broken, or a bad duplicate, skipping.

I sent a tweet to Fedora to hopefully get some feedback. Because this wasn't a super critical issue I've been slacking on troubleshooting as well I will update here and/or provide a new post with more info.

UPDATE: I am increasingly convinced that this is an error with the repository and not something with my server. Check out the following command output:

Nothing marked as out of sync:
# yum distro-sync
Loaded plugins: amazon-id, rhui-lb
No packages marked for distribution synchronization

No problems listed by `package-cleanup`:
#package-cleanup --problems
Loaded plugins: amazon-id, rhui-lb
No Problems Found

`yum check` finds nothing:
# yum check
Not loading "rhnplugin" plugin, as it is disabled
Loading "amazon-id" plugin
Not loading "product-id" plugin, as it is disabled
Loading "rhui-lb" plugin
Not loading "subscription-manager" plugin, as it is disabled
Config time: 0.012
Yum version: 3.4.3
rpmdb time: 0.000
check all

The cache has been cleaned (repeatedly):
# yum clean all
Not loading "rhnplugin" plugin, as it is disabled
Loading "amazon-id" plugin
Not loading "product-id" plugin, as it is disabled
Loading "rhui-lb" plugin
Not loading "subscription-manager" plugin, as it is disabled
Config time: 0.021
Yum version: 3.4.3
Cleaning repos: epel rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-optional rhui-REGION-rhel-server-releases rhui-REGION-rhel-server-rh-common
Cleaning up everything

No orphans:
# package-cleanup --orphans
Not loading "rhnplugin" plugin, as it is disabled
Loading "amazon-id" plugin
Not loading "product-id" plugin, as it is disabled
Loading "rhui-lb" plugin
Not loading "subscription-manager" plugin, as it is disabled
Config time: 0.012
Setting up Package Sacks
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/supplementary/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/extras/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/supplementary/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rhscl/1/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/rhui-client-config/rhel/server/7/x86_64/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rhscl/1/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rhscl/1/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/extras/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/optional/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/optional/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/supplementary/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/debug
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/optional/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/extras/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/os
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/source/SRPMS
mirrorlist: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os
pkgsack time: 0.327
rpmdb time: 0.000
atomic-release-1.0-19.el7.art.noarch

By default, EC2 instances automatically repopulate mirrorlist URLs configured in /etc/yum.repos.d/*.repo files using the region in which the instance is hosted, like this:

mirrorlist=https://rhui2-cds01.REGION.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/$releasever/$basearch/os

I've manually updated the relevant .repo file with my region and upped the debugging level variables for yum-cron to try to narrow things down a bit. No answers yet ...

LATEST UPDATE (11-19): I believe I somewhat figured this out quite a while ago, but I just haven't had the time to update this post.

Amazon manages the licensing information for EC2 instances with operating systems that require it - like Windows and RHEL. So, the short answer is: Amazon broke it. I can't remember off-hand what the licensing agreement is in relation to this particular issue. I do know that I was still paying the exorbitant monthly rate for an RHEL-licensed instance. And I certainly received no notification that my RHEL license was expiring.

This was a very bad experience. The fact is, there are very few reasons why a non-enterprise scale user would ever use RHEL as opposed to CentOS. For Enterprise users that do require licensing, I would highly recommend looking into a Satellite-based updating solution. I'm not sure ATM what the logistics of doing such a thing using a platform like Amazon, but I am sure to be doing my homework on the subject shortly.

Thursday, November 13, 2014

I Ran Windows 7 Updates and My Desktop Went Completely Black! What Do I Do?!

So last night (11-12-14) or this morning you ran a package of `Important` Windows Cumulative Security Updates. Gee those do sound important! There were about 11 or so - specifically, the ones most likely to give you trouble are these:

Update for Windows 7 for x64-based Systems (KB3008627)
Security Update for Windows 7 for x64-based Systems (KB3003743)
Security Update for Windows 7 for x64-based Systems (KB2993958)
Security Update for Windows 7 for x64-based Systems (KB2991963)
Security Update for Windows 7 for x64-based Systems (KB3005607)
Security Update for Windows 7 for x64-based Systems (KB2992611)
Security Update for Windows 7 for x64-based Systems (KB3010788)
Security Update for Windows 7 for x64-based Systems (KB3002885)
Security Update for Windows 7 for x64-based Systems (KB3006226)


After diligently downloading and installing these updates, you allow your computer to reboot. The boot process goes smoothly, you log into your computer, only to find a stark black screen greeting you. Your entire intricately-designed array of desktop icons is gone. Your Desktop Image is replaced by an inky black nothing. Worse even than the blackness of space - even space has stars.

The frank obituary to your beautiful desktop's demise is the following: 

C:\windows\system32\config\systemprofile\Desktop refers to a location that is unavailable. It could be on a hard drive on this computer or on a network. Check to to make sure the disk is properly inserted.

And its' not just the icons or the desktop. Trying to search for an item from the Start menu will produce an error along these lines (where searchstring is whatever you typed in the taskbar): 

"Windows cannot find `search:query=searchstring` Make sure you typed the name correctly, and then try again." 

Microsoft's Mouse and Keyboard Center failed to load completely; despite this, my laptop's USB mouse and embedded touch pad functioned properly.

Even non-Windows related applications will have problems. When I encountered this error, I had to launch Google Chrome as an Administrator in order to get it to run. I also use an incredibly handy text editor in Windows called Notepad++. Notepad++ is an ingeniously formatted gem of a Windows text editor; it can open text files that are sized well into the tens of megabytes without crashing, it color encodes scripted text for programming; its awesome. Use it; its free. Anyway, launching Notepad++ also produced errors; the application was unable to find a variety of XML configuration files.

We've established that this problem sucks. So how do we fix it?

First, you may have problems opening a Command Prompt due to the `search:query` error mentioned above. If you have Powershell installed, use that - it will save time and headaches and can function exactly as a normal command prompt would. If installed you can typically find Powershell in the Start Menu by navigating to All Programs -> Accessories -> Windows Powershell

If you do not have Powershell installed you will have to suffer through by opening a window from `My Computer` by navigating through the Start Menu: click Start and then Computer. It is likely that each time you open a window in this manner a new error message will be produced telling you that your systemprofile\Desktop is missing. You can ignore the error, clicking OK to remove it and proceed.

From here on we will be using in example in which the username we are using is Josh. On your computer you will of course replace Josh with your own username.

Either way you use (powershell of window), navigate to the User directory, which for the purposes of this tutorial will be C:\Users\Josh

We are here to check first and foremost that your actual Desktop folder and files still exists. Click or `cd` to the Desktop folder and take a quick look to ensure that everything still exists. If it does, proceed with the tutorial. If your Desktop folder is missing, than stop here - the issue I am describing should not have caused the entire deletion of your Desktop. You will need to restore these files from backup before continuing to troubleshoot; I hope you kept a backup!

Anyway, for those of us who found that C:\Users\Josh\Desktop exists and is populated with files, we will then navigate to the root of the problem: C:\windows\system32\config\systemprofile\

In this directory you are likely to find three items: Two folders, one named "AppData" and the other named "Contacts". The third item will likely be ntuser.dat - although it may be missing if your Windows folder settings are configured to "hide protected operating system files".

FYI don't be a wimp - BE A POWER USER and go to Organize -> Folder and Search Options -> View tab. From there UNclick "Hide protected operating system files" and select the radio button next to "Show hidden files, folders, and drives". Once you have done this you will notice that a new universe of system files is now available for your perusal. I offer less experienced users this tidbit with the explicit promise that they will refrain from two things:

        1. DO NOT Delete Files Because You "Don't Know What They Do".
             Only Delete Files That You Fully Understand.
        
        2. When You Encounter an Esoteric File DO NOT Search the Name of That File in Google. 
             All of the Websites in Google Will Tell you it is a Virus and Compel You to Purchase Their
             Magical Program to Remove Said Virus. To Understand System Files, You Must RTFM and
             Other Actual Books. Like From a Library Books.

Anyway back to the fix. Within the directory C:\windows\system32\config\systemprofile\  you must create a new directory (by right-clicking and selecting New -> Folder or issuing the command mkdir in PowerShell) and name it "Desktop".

Immediately after creating this directory you may notice that some icons have appeared on the black-as-death desktop. However, these won't include your normal icons, and your desktop image as well as any items you have stuck to the taskbar will not have reappeared. Thats okay - right now you are relying on a broken copy of the "Public" Desktop, and the appearance of icons is a signal that the Public profile is getting better.

To finally get all of your profile settings back, along with the precious icons and desktop doo-dads, simply goto Start, hover your mouse over the arrow next to "shut down" (not shut down, just the arrow) and click "Log off" from the resulting contextual menu.

You will be prompted to Log back into your account. Do so, and you will find that everything in your desktop is back to normal. Enjoy!

RAT Bastard

Earlier this week, several servers I maintain were targeted by automated attempts to upload a remote access trojan (RAT). The RAT is a simpl...