Sunday, January 23, 2011

HMC - Enough to get by on


The Hardware Management Console (HMC) runs a modified Linux operating system and system management software. Technical support for the HMC is provided by IBM's pSeries support center. The HMC is a closed system. Only IBM-approved software is allowed to run on the HMC. Normal Linux errata (including security APAR's) should not be installed on the HMC. Custom configurations and Linux system settings cannot be altered. Therefore, it is not possible to meet the IBM Security requirements that are documented in the technical specification for Linux. To ensure warranty and contract adherence, do not install any non-IBM-approved software or make any configuration changes that are not documented in the HMC users guide.  
                Version 3 is specific to Power 4+ pseries frames, p690, p670, p650, etc
                Version 4 is specific to Power 5 pseries frames all p5##.


Security

While the HMC is considered to be somewhat of a hardware appliance by IBM, the following security controls have been put into place by the UNIX support team to ensure the basic security controls as prescribed by ITCS104 are covered.

HMC Registration and system activation:  As the HMC requires a network connection in most cases for the purpose of remote pSeries support and hardware management, the HMC should always be connected to the IBM Blue Zone Intranet. HMC systems should not be connected to any customer, private, DMZ, or Internet facing network. The HMC should be registered in the MaD database and scanned periodically to ensure that network security vulnerabilities are made known.

HMC Security Fixes: The IBM pSeries support center releases corrective service and security advisories periodically. System administrators and HMC users can subscribe to the IBM Fix Central security advisory center via the HMC website listed above. The Security Administrator within the department as long as the HMC expert subscribe to the Advisory mailing list. Corrective servive and security patches are installed on the HMC's systems supported by the department as needed. Changes to the HMC's are tracked using the IBM America's change and problem management process just like other systems supported by the department. The CIRATS database is used to keep track of resolvable security noncompliance issues for the HMC systems.

HMC root and hscroot userid passwords: Are managed via the same process for managing the root password for other systems supported by the department.

HMC through firewalls (via WebSM) uses a number of ports
port 22 for ssh
port 80
port 9090 for initial connection
from 1 to 3 ephemeral ports in the range 1024-65535 for ongoing communication
Note: In the 520 release, we added the capability to change both the initial connect port and the secondary port.  If you are using WebSM behind a firewall, you will most likely need to change the secondary port range to be a fixed range.
To set the range for the secondary port, you need to run the comand:
 /usr/websm/bin/wsmserver -enable -portstart port1 -portend port2
Where port1 and port2 specify the range of ports to use.  For example, if you only wanted to use the ports 20000 through 20010, you would run:
/usr/websm/bin/wsmserver -enable -portstart 20000 -portend 20010
This would make the secondary communication port come from the inclusive range 20000-20010 instead of being a random port. 
Changing the secondary port does not affect the initial connection port of 9090.  To change the initial connection port, you would have to use the command:
 /usr/websm/bin/wsmserver -enable -listenport connectport
Where connectport is the port for the initial connection.  For example, to change the initial connection port to be 10000 instead of 9090, run
the command:
/usr/websm/bin/wsmserver -enable -listenport 10000
If you want to change both the initial connection port and the secondary connection ports, you have to set both with the same wsmserver command. For example:
/usr/websm/wsmserver -enable -listenport 10000 -portstart 20000 -portend 20010
One important thing to remember is that if you change the initial connection port to be something other than 9090, you need to change the
way you specify hosts in the WebSM console.  When the initial connection port is different, you must specify the hostname as hostname:port.  For example:
mysystem:10000
If you just specify the host as 'mysystem', the WebSM client will attempt to connect to port 9090.  So it is best to not change the
initial connection port if you don't have to.  Just open port 9090 in the firewall so you don't have to give the port number with the host.
So the best thing to do if you are running WebSM behind a firewall is to just set the port range with -portstart and -portend and open that port
range in the firewall along with 9090.
For HMC, since the WebSM server is a service started under xinetd, the /etc/xinetd.d/websm file will need to modified to apply the port
configuration settings.



Remote Access to the HMC via WebSM and SSH is not enabled by default at the time of Install.
            At the HMC console login with hscroot and enable both WebSM and SSH by selecting;
            HMC Management
            HMC Configuration
            Customize Network Settings
                LAN Adapters tab, select the adapter configured for the Blue Zone / 9. net, for version 4 HMCs this will be eth1 and select Details
                Firewall tab select WebSM, Secure Shell and the Allow Incoming button.

Install WebSM on your Windows or UNIX  workstation.
                Using your web browser of choice (Internet Explorer or Netscape)
                Access a HMC using the address format:             http://hmchostname/remote_client.html
                                Replace the above 'hmchostname' with the actual HMC hostname or IP address.
                                An example is:   http://test.test.com/remote_client.html
                                Login with your userid and password       
                Select WebSM installation of your choice and follow the instructions.

Remote Access via WebSM
                Launch the WebSM application.
                Enter the FQDN of the HMC followed by the tab key on the Log On Panel.
                                Allow the handshake to complete before continuing.
                Enter your userid userid and password.
                                The password for the userid 'hscroot' should be the same as our root user standard.

Remote Access via SSH
                # ssh -l your_userid@hmchostname/ip/dns entry.
                                               

Notes:                 hscroot password MUST be changed in the gui, command line change does not update the object database.
                                In our database under the Server By Account view, HMCs are listed with the OS type as Linux and the version as RedHat HMC
                                All of the typical frame management functions are available through the single HMC desktop or WebSM.





Cross-certify ssh from root@codeman to hscroot@HMC
For automation, we cross-certify ssh from root@codeman to hscroot on each HMC:
On codeman:
mykey=`cat $HOME/.ssh/codeman.pub`
ssh hscroot@YOURHMC mkauthkeys -a \"$mykey\"
respond to the hscroot password prompt
Now ssh from root to hscroot@YOURHMC and it should not prompt you for a password


Set up time sync
chhmc -c xntp -s add -a <time server>
hhmc -c xntp -s enable



            Web SM / GUI
                Login to the HMC at console or with WebSM.
            Drill into and right click on  the specific Partition and select Open Terminal Window - be patient.
                Note:    If the terminal window opens but is not displaying anything like the usual SMS screens or a login prompt then AIX has locked up like it would when running out of paging space.
                                The terminal window will identify which partition it is for in the top bar of the window.
                                The State and Operator Panel Value (LCD) do indicate the state of the partition on the Server Management screen
                                Right click the partition, select Operating System / Reset. Be sure you right click the correct partition.
crtl-Ins # cuts in vterm
shft-Ins # pastes in vterm

            HMC / SSH
            SSH into the HMC using the hscroot userid
                        # ssh hscroot@hmchostname     or ip address
                                $ /opt/hsc/bin/vtmenu
                The tool will retrieve and display a list of the LPARS.
                Select the # of the one you need and you will be presented with a console login prompt.

                After you exit the LPAR;
                                ~.            to close vtmenu, (ends your ssh session into the HMC also)
                                    vtmenu # lists all the lpars select # of the console you want ~. to exit
                                    ~. closes Secure IT
                                    ~~. closes the conection to the HMC
                                    ~~~. closes the terminal to the lpar / returns you to vtmenu




                Note:    The below is as provided by support but has not been confirmed.
                                                ~. above should only kill the vtmenu session
                                                mkvterm -m <mged_system> -p <partition>                                                             might fix it.
                                                /opt/hsc/bin/query_cecs                                                                                                will return the managed systems
                                                /opt/hsc/bin/query_partition_names          -m <managed systems>                will return partition names.

                       




                In order for dynamic allocation to work you must have network connectivity between the HMC and the LPAR.  If the HMC and the LPAR are separated by a firewall, you must have port 657 open bi-directionally from any effemeral port on either the HMC or the LPAR. You must also be at AIX level 5.2 or greater.

Dynamically Allocate Resources
                Log into the HMC at the console or via WebSM
                Select Server and Partition - Server Management and select the desired "Running" partition.
                Select "Selected" on the tool bar.
                                If Dynamic Logical Partitioning is grayed out the daemons are not running on the selected LPAR.
                Or right mouse button click on the partition name.
                                If Dynamic Logical Partitioning is not present the daemons are not running on the selected LPAR.
                Select Adapters, Processors or Memory
                                Adjust as needed and select OK.
                                                The Working window should appear and indicate Success when complete in less than a couple of minutes.
                If this fails you may need to rmdev the parent and child devices, error text from the above will tell you which parent to rmdev.
                You may need to run cfgmgr on the LPAR.

Check the DLPAR Daemons
                On the LPAR, start as needed.
                #  lssrc -a |grep rsct              should list
                                ctrmc                                     rsct                         53618    active
                                IBM.ERRM                          rsct_rm                  97566    active
                                IBM.ServiceRM                 rsct_rm                  105374  active
                                IBM.CSMAgentRM           rsct_rm                  102772  active
                                IBM.AuditRM                      rsct_rm                  35260    active
                                IBM.HostRM                       rsct_rm                  58914    active
                                IBM.DRM                             rsct_rm                  77616    active
                                ctcas                                     rsct                         31122    active



                       
                        This requires a recovery CD to boot from of the intended version.  Current versions must be requested on CD, see the below section Obtain HMC Recovery CDs
                        You will need to reboot the HMC and have the cd in the HMC so all these steps are at the console.

                        Login with hscroot at the HMC console and select Licensed Internal Code Maintenance, HMC Code Update, Save Upgrade Data, Hard drive. Allow this to complete.

                  Place HMC Recovery CD #1 , Exit and select Reboot.

                        The upgrade panel will come up and ask about a new install or upgrade, F1 - upgrade, F1 a second time.

                        The upgrade will continue and take several minutes, DVD drive will open and it will reboot when complete.
                        and there are some more prompts to answer...I'll get them on the next upgrade.

Note:            If the HMC comes up and does not know itself, (hscroot password at the default abc123, no network, no profiles) and you are sure you did the "Save Upgrade Data, Hard drive" above then your upgrade suffered a known problem.  You will have to call support to get a temporary password and tell them the correct / system reported serial number from the given HMC.  On a command line do # lshmc -v and see the SE# line.  Support will have you do something like this;
                                        Login as hscroot
                                        Create a hscpe userid and set it's password,  same as our root standard works here.
                                        Logout and back in as hscpe
                                        Right the desktop, select Termial, rshterm
                                        # pesh <serial number of the hmc as provided to support and returned from the lshmc -v>
                                        # su -    (su - root with root's pw)
                                        # mount /mnt/upgrade               (look for a doRestore file zero bytes, likely isn't there)
                                        # touch /mnt/upgrade/doRestore      (might pay to look for this file before touching.
                                        # shutdown -r now        (make sure there is nothing in the DVD drive)
                        There will prompts to answer about keeping the NIC config for eth0 and others if present, be sure to answer these prompts correctly.



               
This is Corrective Service Installation only if you are upgrading you will need to do the Software Upgrade first
Download the current update file from ;
                               
Log into the HMC either at the console or via WebSM and select
                v3 Software Maintenance, HMC (note the current version)
            v4 Licensed Internal Code Maintenance, HMC Code Update (note the current version), Install Corrective Service

With a writeable DVD in the the HMC's DVD drive select
                v3/4 Backup Critical Console Data.  This will take a while, just let it complete.

On the same menu, select
                v3/4 Save Upgrade Data, and follow prompts to save the data to the hard drive.
                Failure to do this step could cause the loss of your Network, Async and Service Agent configuration. This data will be retrieved and reapplied after the upgrade completes.

On the same menu, select
                                v3/4 Install Corrective Service

Select the radio button    Download the corrective service file from a remote system, and then apply the downloaded   service file.
                                Remote site    resposity_server
                                Patch file           /inst.images/HMC/v#/maintfile..zip
                                User ID           userid                    must have ftp read access to the /inst.images/HMC suddirs and files.
                                Password         password            must be valid for the given user.
Note:    The above is FTP based and you will not be given any chance to drill into the correct directory or file, so be accurate.
                If you choose the cdrom option you will need to burn the contents of the .zip file onto a CDRW, do not put the contents of the .zip in a subdirectory on the CD, you will not be given the option to drill into the cd.

Select OK.
                A working window will appear and you can watch the progress.  Successful completion and the need for a reboot should appear.
STOP/READ  Version 4.4.2 has 2 update zip files, it is bad mojo to boot between them go back now and do the 0_2 file.
                                v4 has a update completion panel that allows for a automatic reboot, select the option and OK

To reboot the HMC;
                At the HMC console as you exit, the last panel you are presented with by default references Logout, change this to reboot and select ok.
                If you are remote, WebSM does not offer any option to reboot, on exit or anywhere in the tool.  With SSH enabled you can SSH into the HMC and reboot it.
                # ssh -l hscroot@hmchostname
                $ hmcshutdown -r -t 1                                shutdown with restart in 1 minute, you will get the command line back and have the option to exit, the -t option is not required.
                $ hmcshutdown -t now -r                          Shutdown and reboot immediately.
                $ exit








Frequent Question:

How can I turn off the amber Attention Light in the
operator panel?  (I am assuming that you've already
verified that there are no actionable and outstanding
service events.)

Using the HMC:  (Recommended)

   > Service Applications
     > Service Focal Point
       > Service Utilities
         > Highlight the Managed System name

           <Left Click>  Selected
           <Left Click>  System Attention LED
           <Left Click>  Action

Or; if HMC-less, from the command-line of the LPAR
    w/Service Authority

     # /usr/lpp/diagnostics/bin/usysfault -s normal

Or; if HMC-less, from the command-line of the LPAR
    w/Service Authority

     # diag
       > Task Selection
         > Log Repair Action
           > Select sysplanar0

Or; if HMC-less, from the command-line of the LPAR
    w/Service Authority

     # diag
       > Task Selection
         > Identify and Attention Indicators
           > Set System Attention Indicator to NORMAL



Hardware Configuration
                You may want to create a account specific document to keep track of the CPU, Memory and Adapter allocation. The WebSM does not provide a single view of all the hardware. Use this template as a starting point. (1)


Dealing with the GUI
                1. To change from GUI login to command line login on the HMC press control alt F1.

                2. To get back to the GUI from a command line login type control alt F2.

                3. To reboot the HMC from a command line, su to root and type /sbin/reboot.

Collecting LPAR info from the HMC
                Typically; access the HMC via a putty session,
turn-on logging of the session to a file and
then copy/paste the code:


for MANAGEDSYS in `lssyscfg -r sys -F type_model*serial_num`
do
echo "============MANAGED SYSTEM --> ${MANAGEDSYS}"
for LPAR in `lssyscfg -r lpar -m ${MANAGEDSYS} -F name`
do
echo "            ============LPAR --> ${LPAR} --> CPU resources"
lshwres -r proc -m ${MANAGEDSYS} --level lpar --filter lpar_names=${LPAR}
echo "            ============LPAR --> ${LPAR} --> Memory resources"
lshwres -r mem -m ${MANAGEDSYS} --level lpar --filter lpar_names=${LPAR}
echo "            ============LPAR --> ${LPAR} --> Physical adapters"
lshwres -r io --rsubtype slot -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
echo "            ============LPAR --> ${LPAR} --> Virtual Ethernet config"
lshwres -r virtualio --rsubtype eth --level lpar -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
echo "            ============LPAR --> ${LPAR} --> Virtual SCSI config"
lshwres -r virtualio --rsubtype scsi --level lpar -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
echo "            ============LPAR --> ${LPAR} --> LPAR config"
lssyscfg -r lpar -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
echo "            ============LPAR --> ${LPAR} --> LPAR profiles"
lssyscfg -r prof -m ${MANAGEDSYS} --filter lpar_names=${LPAR}
done
done


                       
HMC Communication ports
HMC Open Port Number/Protocol Application
22/TCP                                                                  Secure Shell
80/TCP                                                                  Web Server
9090/TCP                                                              WebSM initial connection
300000-300009/TCP                                          WebSM Communication
657/TCP                                                                Resource Monitoring and Control
657/UDP                                                               Resource Monitoring and Control


HMC Commands
lshmc –n (lists dynamic IP addresses served by HMC)
lssyscfg –r sys –F name,ipaddr (lists managed system attributes)
lssysconn –r sys (lists attributes of managed systems)
lssysconn –r all (lists all known managed systems with attributes)
rmsysconn –o remove –ip <ipaddr from lssysconn list> (removes a managed system from the HMC)
mkvterm –m {msys} –p {lpar} (opens a command line vterm from an ssh session)
rmvterm –m {msys} –p {lpar} (closes an open vterm for a partition)
Activate a partition
chsysstate –m managedsysname –r lpar –o on –n partitionname –f profilename –b normal
chsysstate –m managedsysname –r lpar –o on –n partitionname –f profilename –b sms
Shutdown a partition
chsysstate –m managedsysname –r lpar –o {shutdown/ossshutdown} –n partitionname [-immed][-restart]

Example 1: To retrieve the HMC Code Level, run the following command:
lshmc -V

Example 2: To retrieve the Managed Systems names, run the following command:
lssyscfg -r sys -F name

Example 3: To retrieve the HMC user profiles available, run the following command:
lshmcusr

Example 4: To retrieve the command usage/help for the lshwinfo HMC command, run the following command:
man lshwinfo

Example 5: To retrieve the current LIC levels for a given Managed System, run the following command:
Note: tttt is the machine type, mmm is the model, and sssssss is the serial number of the managed system. The tttt-mmm*sssssss form must be used if there are multiple managed systems with the same user-defined name.
lslic -t sys -m tttt-mmm*sssssss -F + lic_type-ecnumber-activated_level-installed_level-accepted_level

Example 6: To immediately shut down the HMC console and then restart it, run the following command:
hmcshutdown -t now -r

Port 9090 not listening - can't WebSM into HMC.

cat /opt/ccfw/data/FirewallSettings.ethx-NETAPP-INPUT

Web.name|0.0.0.0|0.0..0.0
SecureWeb.name|0.0.0.0|0.0.0.0
ASM.name|0.0.0.0|0.0.0.0
pegasus.name|0.0.0.0|0.0.0.0
RMC.name|0.0.0.0|0.0.0.0
FCS.name|0.0.0.0|0.0.0.0
Bobcat.name|0.0.0.0|0.0.0.0
Eclipse.name|0.0.0.0|0.0.0.0
vtty.name|0.0.0.0|0.0.0.0
vtty_proxy.name|0.0.0.0|0.0.0.0
i5250.name|0.0.0.0|0.0.0.0
ping.name|0.0.0.0|0.0.0.0
cim.name|0.0.0.0|0.0.0.0
l2tp.name|0.0.0.0|0.0.0.0
SLP.name|0.0.0.0|0.0.0.0
RPD.name|0.0.0.0|0.0.0.0
hwserver.name|0.0.0.0|0.0.0.0
ssh.name|0.0.0.0|0.0.0.0
ntp.name|0.0.0.0|0.0.0.0

Websm is not in the above file.  We peshed in and became root

Added the following entry to the file

WebSM.name|0.0.0.0|0.0.0.0

Rebooted
# hmcshutdown -r -t now


HMC 4 - alpha list of commands
The following HMC commands are available in the restricted shell for HMC Version 4.
HMC CLI commands        Command     Description
bkconsdata              Backs up critical console data
bkprofdata              Backs up profile data configuration
chaccfg                 Changes access control configuration
chcod                   Performs Capacity on Demand operation
chhmc                   Changes HMC's configuration
chhmcusr                Changes HMC user attribute
chhwres                 Changes hardware resource configuration (DLPAR)
chled                   Changes the state of an LED
chsacfg                 Changes Service Agent configuration
chsyspwd                Changes password for a managed system
chsysstate              Changes the state of a partition or managed system
chvet                   Activates the on-demand functions of Virtualization Engine technologies
chdump                  Copies managed system dumps from the HMC to DVD or a remote FTP site
getdump                 Offloads a dump from a managed system to the HMC
hmcshutdown             Shuts down the HMC
lsaccfg                 Displays access control configuration information
lscod                   Displays Capacity on Demand information
lsdump                  Displays available managed system dumps
lshmc                   Displays information about the HMC, such as network configuration
lshmcusr                Displays users on the HMC
lshwres                 Displays hardware resource information
lsled                   Displays LED information
lslic                   Displays Licensed Internal Code levels
lsrefcode               Displays reference codes
lssacfg                 Displays Service Agent configuration information
lssvcevents             Displays console or serviceable events
lssyscfg                Displays system resource configuration
lsvet                   Displays Virtualization Engine technologies information
mkaccfg                 Creates access control object
mkauthkeys              Adds or removes ssh keys on the HMC
mkhmcusr                Creates a user on the HMC
mksyscfg                Creates a system resource configuration such as a partition
mksysconn               Adds a managed system to the HMC
mkvterm                 Opens a Virtual Terminal session
pedbg                   Provides debug tools for Product Engineering
pesh                    Provides full shell access to Product Engineering
rmaccfg                 Removes access control object
rmhmcusr                Removes a user on the HMC
rmsyscfg                Removes a system resource configuration such as a partition
rmsysconn               Removes or resets a connection with a managed system
rmvterm                 Closes a virtual terminal session
rsthwres                Restores hardware resource configuration
rstprofdata             Restores profile data
startdump               Starts a managed system dump
updhmc                  Updates code on the HMC
updlic                  Updates Licensed Internal Code on a managed system
viosvrcmd               Issues a command to a virtual I/O server partition
Linux commands for the restricted shell
The following UNIX (Linux) commands are also available in the restricted shell for HMC Version 4
Linux CLI commands Command name     Command name               Command name
basename    cat         clear
cp          cut         date
diff        du          echo
egrep       expr        fgrep
getopt      grep        head
host        less        ls
man         more        mount
netstat     ping        scp
sed         sleep       sort
ssh         sum         tail
umount      uname       who
whoami


Using the HMC to create a detailed "SystemPlan"
A SystemPlan gives details of resources allocated to all LPARs (CPUs,
memory, physical I/O adapters and Virtual I/O adapters) on a System p
server. A SystemPlan can be created on HMC using 'mksysplan ' command
and view it using a browser on Windows workstation. This will be a
useful reference document for System Administrators and as well Server
Architects.

A SystemPlan can also be deployed from another server to duplicate
the environment. This option is available in the HMC GUI screen for
SystemPlans.



Your HMC must (already) be configured for remote access. 

Steps to generate and view a systemplan  below.

1. Login to HMC using ssh command line.

2. Run the command:  mksysplan -f <filename.sysplan> -m <Managed Server>

3. On a  Windows workstation, launch WebSM, connect to HMC and login.

4.  Click on System Plans/Manage System plans.

5. The pop-up window shows the systemplan you created.

6. Click on the systemplan and then click on View.

7. If Step #6 launches a browser window that fails with an error message,
   change the URL as follows:

   - Change http to https

   - Change port 4411 to 9443.

   - Leave the rest of the info. as-is.  Move the cursor to the end and press
     <Return>. You will see the Systemplan.

   - The defect for port 4411 in HMC 5.2 is fixed in 5.2.1 or 6.x code

1 comment: