Translate

Image of Modern Operating Systems (3rd Edition)
Image of XSLT 2.0 and XPath 2.0 Programmer's Reference (Programmer to Programmer)
Image of Advanced Programming in the UNIX Environment, Second Edition (Addison-Wesley Professional Computing Series)
Image of RHCE Red Hat Certified Engineer Linux Study Guide (Exam RH302) (Certification Press)

Exploring Cisco UCS Emulator Internals

Cisco UCS (Unified Computing System, codenamed Project California during development) was introduced in 2009. It was originally developed as a data center server technology optimized for VMware virtualization workloads, but nowadays is used in many mid-size and large enterprises.

One of the key advantages of UCS from a system administration perspective is the radical reduction in system management points to a single management point called the UCS Manager (UCSM) which is implemented as an NX-OS kernel module in a Cisco Nexus 6000 series Fabric Interconnect (FI) switch. A FI is essentially a Top-Of-Rack switch as far as UCS is concerned.

The core of UCSM is the Data Management Engine (DME) which is persisted in an XML database. A DME uses a number of Application Gateways (AG) to talk to the managed physical aspects of the system–server baseboard (CIMC – Cisco Integrated Management Controller), supported LAN and SAN devices (CNAs and HBAs), fabric subsystems (IOM/FEX), and the FI itself.

The UCS Platform Emulator (UCSPE) was developed by Cisco to enable the use of Cisco UCS Manager and the UCS XML API without requiring actual physical hardware. Amongst other uses, it enables users to become familiar with the UCSM GUI, troubleshoot UCS configuration problems, and provides a convenient way to demonstrate UCS operations for training purposes.

This post looks under the hood at the internals of the UCSPE and documents a number of interesting things. It is assumed that you are familiar with UCS if you are reading this post and therefore no attempt is made to explain, UCS, UCSM, service profiles, the underlying UCS architecture or UCS managed objects.

First of all, here are my VMware Workstation virtual machine settings. The only changes I have made to the downloaded OVA is to change the network settings for the three interfaces shown from NAT to host-only and remove the floppy drive.

After starting the VM, you can see UCSPE is running on some sort of a Red Hat or downstream derivative:

What is displayed above is the classic Red Hat SysVinit scripts output.

When we get root access (how is explained later in this post), from /proc/version and the install log, we determine the OS is a cut-down version of CentOS 4.4.3 (see tiny-centos.sh below) running on Linux kernel v2.6.9 with SELinux disabled. The default runlevel is 3 and no GUI is available except via documents and Perl-based CGI applications served by an Apache web sever.

# cat /root/tiny-centos.sh

#!/bin/sh
#
#       Copyright (c) 2007 Owl River Company
#       info@owlriver.com
#       614 - 488 6954
#
IAM=`id -u`
[ "x$IAM" != "x0" ] && {
        echo "Error: only the root user may run me with effect" 1>&2
        exit 1
        }
VERBOSE=""
# VERBOSE="y"
[ "x$1" = "x-v" ] && {
        export VERBOSE="-v"
                shift 1
                }
#
DIRTY="1"
while [ "x$DIRTY" = "x1" ] ; do
export DIRTY="0"
echo "restarting ... " | logger -p local1.info
#
#       anaconda wants bc
for i in `rpm -qa --qf '%{name}\n' | grep -v joe | grep -v ssh | \
        grep -v rsync | grep -v dhclient | grep -v mouse | \
        grep -v sudo | grep -v gpg-pubkey | grep -v lftp | \
        grep -v sel | grep -v policy | grep -v nscd | \
        grep -v kernel | grep -v dos | grep -v ^bc | \
        grep -v grub | grep -v yum |grep -v vim |grep -v iptable`; do
        [ "x$VERBOSE" = "x-v" ] && echo "testing $i" 1>&2
        echo "testing $i" | logger -p local1.info
        [ "x$i" != "x" ] && rpm -e $i 2> /dev/null && {
                [ "x$VERBOSE" = "x-v" ] && echo "   removed $i" 1>&2
                echo "   removed $i" | logger -p local1.info
                export DIRTY="1"
                }
done
done
[ "x$VERBOSE" = "x-v" ] && echo "done ..." 1>&2
echo "done ... " | logger -p local1.info

Here is what the initial console CLI looks like:

Use the config login to enter the initial setup mode. This invokes the /bin/config-startup binary which displays a simple menu.

Here I configure the three UCM IPv4 management interface addresses (eth0, eth1, and eth2):

The first IPv4 address is for Fabric Interconnect A (primary by default), the second IPv4 address is for Fabric Interconnect B (subordinate by default) and the third IPv4 address is for the virtual UCM address as the default setup assumes UCSM is configured in high availability (HA) mode.

This is what you will see if you have Java Runtime Environment 1.6 or later installed (and correctly configured) and point your browser to 192.168.100.12.

This webpage is rendered by a Perl-based MVC (Model View Controller) web framework called Catalyst which uses APE (Ajax Push Engine). The “Launch KVM Manager” button works but a working KVM is crippled in the emulator. All the code for a working KVM is actually present (see kvm.zip) but is not used. The Java-based KVM technology appears to come from Avocent, which is now a division of Emerson Network Power.

This is what you will see initially when you click on the “Launch UCS Manager” button and successfully login to the downloaded Java applet:

Note that any combination of username and password seems to work – admin/admin, cisco/cisco, murphy/murphy – even though admin is the only predefined locally authenticated user.

UCSM is a Java application which is started from a web browser. A JNPL (Java Network Launching Protocol) configuration file (ucsm.jnpl) is first downloaded and executed. Here is the JNPL file for my UCSMPE.

<?xml version="1.0" encoding="UTF-8"?>
<jnlp spec="1.0+" codebase="http://192.168.100.10/ucsm" >
  <information>
    <title>Cisco Systems UCS Manager (UCSPE-192-168-100-12)</title>
    <vendor>Cisco Systems, Inc.</vendor>
    <description>Cisco Systems UCS Manager</description>
    <homepage href="../index.html" />
    <icon kind="default"  href="Nuova.32.png" width="32" height="32"/>
    <icon kind="shortcut" href="Nuova.16.png" width="16" height="16"/>
    <icon kind="shortcut" href="Nuova.24.png" width="24" height="24"/>
    <icon kind="shortcut" href="Nuova.128.png" width="128" height="128"/>
    <icon kind="shortcut" href="Nuova.64.png" width="64" height="64"/>
    <icon kind="splash"   href="splash-centrale.png"/>
    <shortcut>
        <menu submenu="Cisco Systems UCS Manager"/>
    </shortcut>
  </information>
  <security>
   <all-permissions/>
  </security>
  <update check="always" policy="always"/>
  <resources>
    <j2se version="1.6+" href="http://java.sun.com/products/autodl/j2se" java-vm-args="-Dsun.java2d.d3d=false -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=256m"  initial-heap-size="128M" max-heap-size="768M"/>
    <!--<jar href="signed/@centrale-core@" main="true"/>-->
    <jar href="signed/centrale-min.jar" main="true"/>
    <jar href="signed/cres.jar" />
    <!--<jar href="signed/@centrale-gen@" />-->
    <jar href="signed/foxtrot-core-3.0.jar"/>
    <jar href="signed/swingx-1.0.jar"/>
    <jar href="signed/swing-worker.jar"/>
    <jar href="signed/TimingFramework-1.0.jar"/>
    <jar href="signed/OfficeLnFs_2.7.jar"/>
    <jar href="signed/prefuse.2007.10.21.jar"/>
    <jar href="signed/log4j-1.2.15.jar"/>
    <jar href="signed/Stub.jar"/>
    <jar href="signed/jgraph.jar"/>
    <jar href="signed/jna.jar"/>
    <jar href="signed/examples.jar"/>
    <jar href="signed/jfreechart-1.0.9.jar"/>
    <jar href="signed/jcommon-1.0.12.jar"/>
    <jar href="signed/miglayout-3.6-swing.jar"/>
    <jar href="signed/jxlayer.jar"/>
    <jar href="signed/JBusyComponent.jar"/>
    <jar href="signed/jautocomp.jar"/>
    <jar href="signed/lablib-checkboxtree-3.2.jar"/>
    <property name="jnlp.packEnabled" value="true"/>
    <property name="jnlp.ucsm.server.ip" value="192.168.100.10"/>
    <property name="jnlp.ucsm.server.port" value="80"/>
    <property name="jnlp.ucsm.server.port.secure" value="80"/>
    <property name="jnlp.ucsm.log4j.configuration" value="http://192.168.100.10/ucsm/log4j.xml"/>
    <property name="jnlp.ucsm.secure.mode" value="false"/>
    <property name="jnlp.ucsm.sun.awt.exception.handler" value="com.nuova.centrale.core.log4j.NuUnhandledExceptionHandler"/>
    <property name="jnlp.ucsm.centrale.security.enforced" value="true"/>
    <property name="jnlp.ucsm.centrale.login.enforced" value="true"/>
    <!-- property name="jnlp.ucsm.skip.version.check" value="true"/ -->
    <property name="jnlp.ucsm.use.login" value="true"/>
    <property name="jnlp.ucsm.centrale.systemName" value="UCSPE-192-168-100-12"/>
    <property name="jnlp.ucsm.centrale.systemFabric" value=""/>
    <property name="jnlp.ucsm.centrale.systemID" value="35073892-5923-11e4-b901-000c29974df8"/>
    <property name="jnlp.ucsm.centrale.oobIpAddr" value="192.168.100.10"/>
    <property name="jnlp.ucsm.centrale.domains" value=""/>
    <property name="jnlp.ucsm.centrale.clibanner" value=""/>
    <!-- property name="jnlp.ucsm.edge.black" value="false"/ -->
    <!-- property name="jnlp.ucsm.edge.thick" value="false"/ -->
    <property name="jnlp.ucsm.centrale.mainframe" value="app:centrale.3"/>
    <property name="jnlp.ucsm.token" value=""/>
    <property name="jnlp.ucsm.multi.token" value=""/>
    <property name="jnlp.ucsm.initial.naf" value=""/>
    <property name="jnlp.ucsm.fault.severity" value=""/>
    <property name="jnlp.ucsm.fault.type" value=""/>
    <!-- property name="jnlp.ucsm.skip.splash.screen" value="false"/ -->
    <!-- property name="jnlp.ucsm.native.splash" value="false"/ -->
    <!-- Required for w32 window transparency -->
    <!-- property name="jnlp.ucsm.sun.java2d.noddraw" value="true"/ -->
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
  </resources>
  <resources os="Mac OS X">
    <jar href="signed/osx.jar"/>
    <jar href="signed/osx2.jar"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
        <property name="log.show.encrypted" value="true"/>
  </resources>
  <application-desc main-class="com.nuova.centrale.core.NuApp" />
</jnlp>


Technically ucsm.jnlp is generated on the fly by ucsm.cgi from a template named applet.template based on configuration options from ../opt/db/sam.config using a Perl CGI script.

The JNLP protocol, defined with an XML schema, specifies how to launch Java Web Start applications. JNLP consists of a set of rules defining how exactly to implement the launching mechanism. JNLP files include information such as the location of the jar (Java Archive) package file(s) and the name of the main class for the application, in addition to any other parameters. A properly configured browser passes JNLP files to a Java Runtime Environment (JRE) which in turn downloads the application onto the user’s platform and executes it.

By the way, you can launch UCSM and KVM from the command line should you choose to do so. For example, to launch UCSM:

c:\Program Files\Java\jre7\bin>javaws -silent c:\ucs\ucsm.jnlp


assuming a local copy of ucsm.jnlp is in c:\ucs.

You can access the UCSMPE database browser by pointing your browser at <management ip>/visore.html:

Visore is a utility built into UCSMPE that allows a user to easily browse Managed Objects (MOs) using an HTML browser. Nothing can be changed from Visore; it’s usefulness lies in configuration verification.

To really explore the internals of UCSMPE, we need root access to the underlying OS. There are many techniques to do this. The particular technique I used was to boot the UCSMPE VM using Partition Magic, mount /dev/sda1 and modify the GRUB configuration file to add another stanza which, when selected, boots into single-user mode as show here:

Here is what the new GRUB menu looks like when invoked:

When the new menu option is selected, we end use as root in a bash shell:

Here is contents of /etc/passwd:

# cat /etc/passwd
root:x:0:0:root:/root:/bin/apechallenge
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
haldaemon:x:68:68:HAL daemon:/:/sbin/nologin
apeuser:x:500:100::/home/apeuser:/bin/bash
cliuser:x:504:501::/home/cliuser:/apedisk/ape/install/os/clishell.sh
#stamp:x:502:502::/home/stamp:/apedisk/ape/install/os/stampshell.sh
config:x:503:503::/home/config:/bin/config-startup
ntp:x:38:38::/etc/ntp:/sbin/nologin
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin


Note what is invoked when a user successfully logs in as root. Not what you would normally expect to see.

From here, you can use the passwd command to change the password for apeuser and root. To speed up UCSPE startup, you can also turn off the nfs, nfslock and ntpd services using the chkconfig utility. I also usually add an extra user account called toor with a UID an GID of 0 and modify /etc/ssh/sshd_config to permit root and toor access via ssh. Also I suggest you delete the useless root crontab entry relating to ntp which is invoked every minute.

When you have made the above changes, reboot UCSPE and you will have easy access to UCSPE internals from the console or via ssh.

You will see various references to Nuovo when exploring UCSPE internals. In August 2006, Cisco paid $50 million to acquire an 80% stake in Nuova Systems, a Cisco “spin-in”, which was founded by a number of former Cisco senior employees including former senior vice president and chief development officer Mario Mazzola, and Ed Bugnion, a co-founder of VMware. In late 2008, Cisco purchased the remaining 20%. The Nexus 5000 series switch, for example, was a collaborative development between Cisco and Nuova Systems.

LVM is used to provide the /ape filesystem. This is where the majority of interesting stuff lives.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2               901252    589720    265740  69% /
/dev/hda1               124427      9125    108878   8% /boot
none                   1037948         0   1037948   0% /dev/shm
/dev/hdd1              1027768    361308    614252  38% /home
/dev/ram0               247919     84011    163908  34% /mnt
/dev/mapper/ape       10320108   3014080   6781796  31% /ape


It looks like /ape is configured from files in /home/apeuser.

# ls -l /home/apeuser
total 343544
-rw-r--r--  1 apeuser users 331319924 Sep 15 17:24 apedisk.tar.gz
lrwxrwxrwx  1 apeuser users        16 Feb 15  2012 opt -> /apedisk/ape/opt
-rw-r--r--  1 root    root   20109611 Sep 15 17:24 perl.tar.gz
-rw-r--r--  1 root    root       1594 Sep 15 17:24 run_once_inside_vm_distro.sh
lrwxrwxrwx  1 apeuser users        20 Feb 15  2012 tmp -> /apedisk/ape/var/tmp

# cat /home/apeuser/run_once_inside_vm_distro.sh
#!/bin/sh

# as in the name: download this script along with apedisk.tar.gz into /home/apeuser inside the VM
# run it once as root. After this, VM is ready to ship, so shut it down with 'shutdown -h now'
# and once vmplayer exits, run vmzip.sh against the VM directory.

umount /mnt
mkfs.ext2 -m 0 /dev/ram0
mount /dev/ram0 /mnt

cd /mnt
chown apeuser.users /home/apeuser/apedisk.tar.gz

tar xzvf /home/apeuser/apedisk.tar.gz   \
    ape/install/os/init.d/cisco_ucspe   \
    ape/install/os/init.d/init_ip       \
    ape/install/apevmsec                \
    ape/install/config-startup          \
    ape/install/os/config-sudo

cp -f ape/install/os/init.d/cisco_ucspe /etc/init.d
chmod 744 /etc/init.d/cisco_ucspe
cp -f ape/install/apevmsec /etc/init.d
chmod 744 /etc/init.d/apevmsec
cp -f ape/install/os/init.d/init_ip /etc/init.d
chmod 744 /etc/init.d/init_ip
mv -f ape/install/config-startup /bin
chown root.root /bin/config-startup
chmod 755 /bin/config-startup
mv -f ape/install/os/config-sudo /bin
chown root.root /bin/config-sudo
chmod 700 /bin/config-sudo
cd /

umount /mnt
# /opt is not used, but if exists, it causes a "disk usage exceeded 70%" fault in UCSM
rmdir /opt

cd /etc/rc3.d/
ln -sf ../init.d/cisco_ucspe S96cisco_ucspe
cd ../rc0.d/
ln -sf ../init.d/cisco_ucspe K05cisco_ucspe
cd ../rc6.d/
ln -sf ../init.d/cisco_ucspe K05cisco_ucspe

# deny ssh to anyone but root
echo "AllowUsers root cliuser" >> /etc/ssh/sshd_config

# doesn't work yet
#cd /etc/rc3.d/
#ln -sf ../init.d/init_ip S07init_ip

echo "Cisco UCS Platform Emulator 2.2(3aPE1)(391886)" > /etc/redhat-release


Spot the obvious error in the above script?

The last command reveals that this particular version of UCSPE was probably constructed in August 2010.

reboot   system boot  2.6.9-89.0.23.EL Thu Nov 25 09:54          (00:05)
root     pts/0        nmtg-hq-salt1-37 Thu Nov 25 08:57 - down   (00:01)
root     pts/0        nmtg-hq-salt1-37 Thu Nov 25 08:57 - 08:57  (00:00)
reboot   system boot  2.6.9-89.0.23.EL Thu Nov 25 08:55          (00:03)
root     pts/0        nmtg-hq-salt1-37 Thu Nov 25 07:58 - down   (00:01)
reboot   system boot  2.6.9-89.0.23.EL Thu Nov 25 07:57          (00:02)
root     pts/0        nmtg-hq-salt1-37 Thu Nov 25 07:40 - down   (00:16)
reboot   system boot  2.6.9-89.0.23.EL Thu Nov 25 07:39          (00:17)
root     pts/0        nmtg-hq-salt1-37 Wed Nov 24 15:10 - down   (16:07)
reboot   system boot  2.6.9-89.0.23.EL Wed Nov 24 15:09          (16:08)
root     pts/1        nmtg-hq-salt1-37 Wed Nov 24 14:40 - down   (00:27)
config   tty1                          Wed Nov 24 14:38 - 14:38  (00:00)
root     pts/0        nmtg-hq-salt1-37 Wed Nov 24 14:00 - 15:07  (01:07)
reboot   system boot  2.6.9-89.0.23.EL Wed Nov 24 13:58          (01:09)
root     pts/0        savbu-mgmt09.cis Sat Nov 13 22:01 - down   (00:02)
reboot   system boot  2.6.9-89.0.23.EL Sat Nov 13 22:00          (00:02)
root     pts/0        savbu-mgmt09.cis Wed Nov 10 00:29 - down   (00:00)
root     pts/0        savbu-mgmt09.cis Wed Nov 10 00:28 - 00:29  (00:00)
reboot   system boot  2.6.9-89.0.23.EL Wed Nov 10 00:27          (00:02)
root     pts/0        savbu-mgmt09.cis Mon Nov  8 23:06 - down   (00:02)
reboot   system boot  2.6.9-89.0.23.EL Mon Nov  8 23:06          (00:02)
root     pts/0        savbu-mgmt09.cis Mon Nov  8 05:32 - down   (00:01)
root     pts/0        savbu-mgmt09.cis Mon Nov  8 05:22 - 05:28  (00:06)
reboot   system boot  2.6.9-89.0.23.EL Mon Nov  8 05:22          (00:11)
root     pts/0        savbu-mgmt09.cis Tue Oct 19 22:03 - down   (00:03)
reboot   system boot  2.6.9-89.0.23.EL Tue Oct 19 22:00          (00:06)
root     pts/0        savbu-mgmt09.cis Sat Aug  7 01:33 - down   (00:05)
reboot   system boot  2.6.9-89.0.23.EL Sat Aug  7 01:33          (00:05)

wtmp begins Fri Aug  6 22:51:56 2010


However the date of /root/install.log appears to indicate August 2007 as the initial installation date.

The document root for the webpages served by UCSPE is /ape/ape/opt/apache/htdocs

# cd /ape/ape/opt/apache/htdocs
# ls -l
total 14308
-rwxr-xr-x  1 apeuser users     3946 Sep 15 14:39 ape.cgi
drwxr-xr-x  6 apeuser users     4096 Sep 15 14:39 bootstrap
-rw-r--r--  1 apeuser users       94 Sep 15 14:26 crossdomain.xml
drwxr-xr-x  2 apeuser users     4096 Sep 15 14:26 css
drwxr-xr-x  6 apeuser users   524288 Sep 15 14:39 docs
-rw-r--r--  1 apeuser users     3638 Sep 15 14:26 favicon.ico
drwxr-xr-x  3 apeuser users     4096 Sep 15 14:39 help
drwxr-xr-x  4 apeuser users     4096 Nov 23 19:23 images
-rw-rw-rw-  1 apeuser users    10160 Nov 23 19:02 index.html
drwxr-xr-x  2 apeuser users     4096 Nov 23 20:06 js
-rw-r--r--  1 apeuser users    10902 Sep 15 14:26 kvm.html
-rw-r--r--  1 apeuser users 11067186 Sep 15 14:26 kvm.zip
drwxr-xr-x  6 apeuser users     4096 Sep 15 14:40 pictures
-rw-r--r--  1 apeuser users   268820 Sep 15 14:42 sample_sel.txt
lrwxrwxrwx  1 apeuser users       42 Nov 23 11:09 static -> /apedisk/ape/webconfig/ape/Ape/root/static
drwxr-xr-x  2 apeuser users     4096 Sep 15 14:40 templates
drwxr-xr-x  2 apeuser users     4096 Sep 15 14:40 tools
drwxr-xr-x  7 apeuser users     4096 Nov 23 20:02 ucsm
drwxr-xr-x  5 apeuser users     4096 Sep 15 14:40 visore
-rwxr-xr-x  1 apeuser users     3798 Sep 15 14:40 visore.html
-rw-r--r--  1 apeuser users  2677560 Sep 15 14:39 xml_Schema_and_Samples.zip

# cd ucsm
# ls -al
total 1088
drwxr-xr-x   7 apeuser users   4096 Nov 23 20:02 .
drwxr-xr-x  13 apeuser users   4096 Nov 23 19:29 ..
-rw-r--r--   1 apeuser users   3579 Sep 15 14:40 applet.template
-rw-r--r--   1 apeuser users   4804 Sep 15 14:39 avocent.jnlp.tmpl
-rw-r--r--   1 apeuser users   3872 Sep 15 14:39 centrale-applet.jnlp.tmpl
-rw-r--r--   1 apeuser users   5691 Nov 23 19:02 centrale.jnlp.tmpl
-rw-r--r--   1 apeuser users 884462 Sep 15 14:39 centralina.swf
drwxr-xr-x   2 apeuser users   4096 Sep 15 14:39 chassis
-rwxr-xr-x   1 apeuser users    635 Sep 15 14:40 domains.cgi
-rwxr-xr-x   1 apeuser users   4400 Sep 15 14:40 getkvmurl.cgi
drwxr-xr-x   3 apeuser users   4096 Sep 15 14:39 help
-rw-r--r--   1 apeuser users    208 Sep 15 14:39 .htaccess
-rwxr-xr-x   1 apeuser users     30 Nov 23 19:02 isSamInstalled.cgi
-rwxr-xr-x   1 apeuser users   3669 Sep 15 14:40 javahelp.jnlp
-rw-r--r--   1 apeuser users   5132 Sep 15 14:39 kvm.html
-rwxr-xr-x   1 apeuser users   3668 Sep 15 14:40 kvm.jnlp
-rw-r--r--   1 apeuser users    529 Sep 15 14:40 kvm_vm_help.html
-rw-r--r--   1 apeuser users   2229 Sep 15 14:40 log4j-applet.xml
-rw-r--r--   1 apeuser users   2381 Sep 15 14:40 log4j.kvm.xml
-rw-r--r--   1 apeuser users   2386 Sep 15 14:39 log4j.xml
-rw-r--r--   1 apeuser users   5957 Sep 15 14:40 Nuova.128.png
-rw-r--r--   1 apeuser users    562 Sep 15 14:39 Nuova.16.png
-rw-r--r--   1 apeuser users    834 Sep 15 14:39 Nuova.24.png
-rw-r--r--   1 apeuser users   1205 Sep 15 14:39 Nuova.32.png
-rw-r--r--   1 apeuser users   2006 Sep 15 14:40 Nuova.64.png
-rw-r--r--   1 apeuser users    657 Sep 15 14:39 playerProductInstall.swf
drwxr-xr-x   2 apeuser users   4096 Sep 15 14:39 signed
-rw-r--r--   1 apeuser users  50910 Sep 15 14:40 splash-centrale.png
-rw-r--r--   1 apeuser users  23467 Sep 15 14:39 swfobject.js
drwxr-xr-x   2 apeuser users   4096 Sep 15 14:40 tools
drwxr-xr-x   3 apeuser users   4096 Sep 15 14:40 ucsm
-rwxr-xr-x   1 apeuser users   2455 Sep 15 14:40 ucsm.cgi
-rwxr-xr-x   1 apeuser users   3690 Nov 23 19:03 ucsm.jnlp


Note that the UCS KVM facility is also launched using a .jnlp configuration file. Also note the two Adobe Shockwave/Flash files. To fully use UCSPE, Adobe Shockwave or Flash must be enabled. The signed directory contains the jar files used by JRE to execute the UCSM application.

Apache uses three modules, i.e. mod_nuova, mod_proxy_http and mod_proxy.

You should read the ape.install, ape.run and ape.shutdown shell scripts to fully understand how UCSMPE is started and stopped.

The configuration menu that is displayed when you login as config invokes the ../os/config-sudo script. Have a read of this file. It contains lots of useful information.

Hardware configuration information is kept in /etc/ape/ape.db which is an SQLite database.

sqlite3 ape.db
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE chassis_config (
        id              INTEGER PRIMARY KEY AUTOINCREMENT,
        external_id             INTEGER, -- a chassis id assigned by UCSM
        name            TEXT,
        type                    TEXT, -- running vs startup
        is_removed              INTEGER DEFAULT(0),
        pid                             TEXT, -- refers to CapProvider:pid sam_default_config.xml
        serial                  TEXT,
        switch_id       TEXT,
        sw_slot_id      INTEGER,
        sw_port_id      INTEGER,
        mux_slot_id     INTEGER,
        mux_port_id     INTEGER,
        power                   INTEGER DEFAULT(0), -- (1) on (0) off
        locator_led             INTEGER DEFAULT(0) -- (1) on (0) off
    );
CREATE TABLE chassis_fan (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        chassis_config_id       INTEGER REFERENCES chassis_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
    );
CREATE TABLE chassis_iom_fi(
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        chassis_id INTEGER DEFAULT(0) REFERENCES chassis_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
        iom_id INTEGER DEFAULT(1),
        port INTEGER DEFAULT(0),
        chassis_external_id INTEGER DEFAULT(0),
        link_id INTEGER DEFAULT(0),
        iom_port INTEGER DEFAULT(0));
CREATE TABLE chassis_psu (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        chassis_config_id       INTEGER REFERENCES chassis_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT, -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
        power                           INTEGER DEFAULT(0) -- (1) on (0) off (whether PSU power cord is connected)
    );
CREATE TABLE chassis_server (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        name                    TEXT,
        chassis_config_id       INTEGER REFERENCES chassis_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        is_removed                      INTEGER DEFAULT(0),
        prev_chassis_id         INTEGER, -- used only in trash can
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        serial                          TEXT,
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT, -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
        locator_led                     INTEGER DEFAULT(0), -- (1) on (0) off
        power                           INTEGER DEFAULT(0) -- (1) on (0) off (whether PSU power cord is connected)
    );
CREATE TABLE fex_cluster(id INTEGER PRIMARY KEY AUTOINCREMENT,switch_A INTEGER DEFAULT(0) REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE ,switch_B INTEGER DEFAULT(0) REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE);
CREATE TABLE fex_config (
        id              INTEGER PRIMARY KEY AUTOINCREMENT,
        external_id             INTEGER, -- a chassis id assigned by UCSM
        name            TEXT,
        type                    TEXT, -- running vs startup
        is_removed              INTEGER DEFAULT(0),
        pid                             TEXT, -- refers to CapProvider:pid sam_default_config.xml
        serial                  TEXT,
        switch_id       TEXT,
        sw_slot_id      INTEGER,
        sw_port_id      INTEGER,
        mux_slot_id      INTEGER,
        mux_port_id      INTEGER,
        power                   INTEGER DEFAULT(0), -- (1) on (0) off
        locator_led             INTEGER DEFAULT(0), -- (1) on (0) off
        switch_side    TEXT DEFAULT('A')
        );
CREATE TABLE fex_fan (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        fex_config_id       INTEGER REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE fex_iom_fi(
       id INTEGER PRIMARY KEY AUTOINCREMENT,
       fex_id INTEGER DEFAULT(0) REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
       port INTEGER DEFAULT(0),
       fex_external_id INTEGER DEFAULT(0),
       link_id INTEGER DEFAULT(0),
       iom_port INTEGER DEFAULT(0),
       iom_side TEXT DEFAULT('A'));
CREATE TABLE fex_psu (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        fex_config_id       INTEGER REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT, -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
        power                           INTEGER DEFAULT(0) -- (1) on (0) off (whether PSU power cord is connected)
);
CREATE TABLE fex_state(id INTEGER PRIMARY KEY AUTOINCREMENT,fex_id INTEGER DEFAULT(0) REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE, fex_id_B INTEGER DEFAULT(0),  state INTEGER DEFAULT(0));
CREATE TABLE rack_adapter (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE rack_adapter_allowed (id INTEGER PRIMARY KEY AUTOINCREMENT,pid_server TEXT,pid_adapter TEXT);
CREATE TABLE rack_cpu (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE rack_cpu_allowed (id INTEGER PRIMARY KEY AUTOINCREMENT,pid_server TEXT,pid_cpu TEXT);
CREATE TABLE rack_details(
pid TEXT PRIMARY KEY,
sockets INTEGER,
rackunits INTEGER,
dimms INTEGER,
hdds INTEGER,
cpus INTEGER,
pcies INTEGER,
psus INTEGER, storage_controllers INTEGER);
CREATE TABLE rack_dimm (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE rack_dimm_allowed (id INTEGER PRIMARY KEY AUTOINCREMENT,pid_server TEXT,pid_dimm TEXT);
CREATE TABLE rack_fex(
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                fex_id INTEGER DEFAULT(0) REFERENCES fex_config(id) ON DELETE CASCADE ON UPDATE CASCADE ,
                fex_id_B INTEGER ,
 adapter_id INTEGER DEFAULT(0) REFERENCES rack_adapter(id) ON DELETE CASCADE ON UPDATE CASCADE,
                fex_port INTEGER DEFAULT(0),
                fex_port_B INTEGER DEFAULT(0),
                swm_config INTEGER DEFAULT(0));
CREATE TABLE rack_fi(
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        adapter_id INTEGER DEFAULT(0) REFERENCES rack_adapter(id) ON DELETE CASCADE ON UPDATE CASCADE,
        fi_port INTEGER DEFAULT(0),
        rack_id INTEGER DEFAULT(0) REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        side TEXT DEFAULT('A'),
        swm_config INTEGER DEFAULT(0));
CREATE TABLE rack_hdd (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE rack_hdd_allowed (id INTEGER PRIMARY KEY AUTOINCREMENT,pid_server TEXT,pid_hdd TEXT);
CREATE TABLE rack_psu (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id       INTEGER REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT, -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
        power                           INTEGER DEFAULT(0) -- (1) on (0) off (whether PSU power cord is connected)
);
CREATE TABLE rack_psu_allowed(
id INTEGER PRIMARY KEY AUTOINCREMENT,
pid_server TEXT,
pid_psu TEXT
);
CREATE TABLE rack_server(id INTEGER PRIMARY KEY AUTOINCREMENT,name TEXT,fex_id INTEGER DEFAULT(0), fex_id_B INTEGER DEFAULT(0), slot_id INTEGER, pid TEXT, serial TEXT,vendor TEXT,model TEXT,locator_led INTEGER DEFAULT(0),power INTEGER DEFAULT(0), type TEXT, is_removed INTEGER DEFAULT(0), lom_fex_port_1 INTEGER DEFAULT(0), lom_fex_port_2 INTEGER DEFAULT(0));
CREATE TABLE rack_storage (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES rack_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE scaled_details(
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        master_id INTEGER DEFAULT(0) REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slave_id INTEGER DEFAULT(0) REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE);
CREATE TABLE server_adapter (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
    );
CREATE TABLE server_cpu (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
    );
CREATE TABLE server_dimm (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
    );
CREATE TABLE server_hdd (
                id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
    );
CREATE TABLE server_storage (
  id                                      INTEGER PRIMARY KEY AUTOINCREMENT,
        server_id                       INTEGER REFERENCES chassis_server(id) ON DELETE CASCADE ON UPDATE CASCADE,
        slot_id                         INTEGER,
        pid                                     TEXT, -- refers to CapProvider:pid sam_default_config.xml
        vendor                          TEXT, -- vendor and model should come from catalogue based on PID, but if during discovery
        model                           TEXT  -- we get an unknown PID, we store vendor/model explicitly (i.e. if these fields are NULL, use pid)
);
CREATE TABLE settings (
        id              INTEGER PRIMARY KEY,
        ha              INTEGER,
        swm             INTEGER,
        das             INTEGER,
       persistence     INTEGER,
        switchType      INTEGER,
startup_cfg_url TEXT            -- URL of a remote config server; APE will load its config from this url, can be provided by another APE VM
       ,iom             TEXT
 , use_dhcp INTEGER DEFAULT(1), ip_addr TEXT, netmask TEXT, gateway TEXT, num_uplinks INTEGER DEFAULT(1), ntp_use_default INTEGER DEFAULT(1), ntp_ip_addr TEXT,version TEXT);
CREATE TABLE smartplay_details(id PRIMARY KEY,
ha INTEGER,
type TEXT,
fi TEXT,
filename TEXT,
swm INTEGER);
CREATE TABLE stamp (
        id          INTEGER PRIMARY KEY,
        cisco_user  VARCHAR(20),
        name        TEXT,
        company     TEXT,
        email       TEXT
    );
CREATE TABLE template_details(id PRIMARY KEY,
pid INTEGER,
description TEXT,
name TEXT,
type TEXT,
filename TEXT);
sqlite>

FI DMEs (Database Management Engine) are also implemented as SQLite databases. For example:

# cd /ape/ape/opt/db/ha/left/flash
# ls -l
total 9040
-rw-r--r--  1 apeuser users 9240576 Nov 23 22:31 dme.db
# sqlite3 dme.db
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE MODB (KEY INTEGER PRIMARY KEY, MOXML BLOB);
sqlite> SELECT * FROM modb LIMIT 8 ;
10002| <aaaLog dn="aaa-log" maxSize="10000" purgeWindow="100" size="450" status="modified"/>
10003| <apeManager dn="ape" statsUpdateId="1" status="created"/>
10004| <callhomeEp adminState="off" alertThrottlingAdminState="on" configState="ok" descr="" dn="call-home" fsmDescr="" fsmPrev="configCallhomeSuccess" fsmProgr="100" fsmRmtInvErrCode="none" fsmRmtInvErrDescr="" fsmRmtInvRslt="" fsmStageDescr="" fsmStamp="1416769439.126" fsmStatus="nop" fsmTry="0" intId="10004" name="" policyLevel="0" policyOwner="local" status="modified"/>
10005| <callhomeAnonymousReporting adminState="off" count="0" dn="call-home/anonymousreporting" sleepInterval="0" status="modified" userAcknowledged="yes"/>
10006| <callhomePeriodicSystemInventory adminState="off" dn="call-home/periodicsysteminventory" intervalDays="30" lastDeadline="0.000" maximumRetryCount="1" minimumSendNowIntervalSeconds="5" nextDeadline="never" pollIntervalSeconds="300" retryCount="0" retryDelayMinutes="10" sendNow="no" status="created" timeOfDayHour="0" timeOfDayMinute="0" timeOfLastAttempt="0.000" timeOfLastSuccess="never"/>
10007| <callhomeProfile alertGroups="all,ciscoTac,diagnostic,environmental" descr="Built-in full text profile" dn="call-home/profile-full_txt" format="fullTxt" level="warning" maxSize="5000000" name="full_txt" status="created"/>
10008| <callhomeProfile alertGroups="all,ciscoTac,diagnostic,environmental" descr="Built-in text profile" dn="call-home/profile-short_txt" format="shortTxt" level="warning" maxSize="5000000" name="short_txt" status="created"/>
10009| <callhomeProfile alertGroups="ciscoTac" descr="Built-in XML Cisco-TAC profile" dn="call-home/profile-CiscoTAC-1" format="xml" level="normal" maxSize="5000000" name="CiscoTAC-1" status="created"/>
sqlite>

Well, this post is getting too long and it is time to stop. By now, you should have sufficient information to explore the internals of the UCS Manager emulator by yourself.

Happy hunting!

Comments are closed.