Site menu:

Site search

July 2009
M T W T F S S
« Oct    
 12345
6789101112
13141516171819
20212223242526
2728293031  

Categories

Tags

Network Statistics

netstat -i 1 (prints out the packet volume on an interface)

iostat -xn 5 (prints out the write rates on disk)

Shutdown and Restart

# shutdown the server in 60 seconds, restart system in administrative state
# (Solaris)
/usr/sbin/shutdown -y -g60 -i1 “System is begin restarted”

# shutdown the server immediately, cold state
# (Solaris)
/usr/sbin/shutdown -y -g0 -i0

# shutdown AIX server, reboot .. also Ctrl-Ctrl/Alt
shutdown -Fr

# restart the server
/usr/sbin/reboot

Detecting 32-bit versus 64-bit operating system

Below is the command which will reveal the nature of your operating system (Solaris only).

echo “Welcome to “`isainfo -b`”-bit Solaris”

# Detect if a binary file is made for 32 or 64 bit systems

file myexe

System Status Check

Listed here are a few system monitoring commands which should give you
a rough idea of how the server is running.

# server information
uname -a

# server config information
prtconf
sysdef -i

# server up time
uptime

# disk free, listed in KB
df -kt

# mounted devices
mount

# network status
netstat -rn

# network configuration info
ifconfig -a

# processes currently running
ps -elf

# user processes
w
whodo
who am i
finger
ps

# virtual memory statistics
vmstat 5 5

# system activity reporter (Solaris/AIX)
sar 5 5

# report per processor statistics (Solaris)
mpstat 5 5
psrinfo

# swap disk status (Solaris)
swap -l

# shared memory
ipcs -b

Veritas Cluster Setup

Veritas Cluster Install Preparation & Planning

This is the MOST important part of the veritas cluster installation process. If you skip a step - you pay for it later!

  • Machines racked
  • Machines jumpstarted or equivalent (last 4 e3500’s needed to be done by hand)
  • On one machine set scsi-initiator to 7
  • Have array installed, verify disks can be seen on both machines
  • Additional patches for veritas installed (download latest sun recommended patches) — these are not in jumpstart!
  • Send off for veritas licemse — if not there in time, get temp license
  • Have software ready for install
  • Hardware meets specifications
  • Veritas checklist should be filled out & ready to go
  • Change internet/network link to qfe1 ; cross over cables on hme0 & qfe0 (REMOVE hostname.home and hostname.qfe0)

Veritas Cluster Install - Day 1

Verify that preplanning & preparation was completed properly. If not, do NOT proceed.

If preparation was completed, on BOTH machines:

    /etc/init.d/volmgt start
    insert cdrom for db edition 2.1 [or higher]
    cd /cdrom/cdrom0
    ./installDBED [answer default/yes to all EXCEPT say no to single / group ]
    [may need to change cdroms here - not sure]
    installvx
    remove cdrom
    vxdiskadm — encapsulate root - specify your 2 root drives
    process will reboot twice
    create mount points for your db

Once done, on ONE machine:

    vxdiskadm - initialize driives in array
    vxassist -g oradg -maxsize drive1 drive2 - setup array drives
    mount disks
    add to vfstab

Veritas Cluster Install - Day 2

DBAs install oracle on ONE machine, update /etc/system on BOTH machines, add table for vcs

Veritas Cluster Install - Day 3

On BOTH machines

    • set-node 0 [on one machine set to 1]
      set-cluster 0
      link hme0 & qfe0
      low-link pri qfe1
      start (at bottom)
      uncomment gabconfig -c -n 2
  • /etc/init.d/volmgt start
    insert cluster server cd
    cd /cdrom/cdrom0
    pkgadd -d . (add packages 3,2,5,1,4,6; yes to everything)
    eject cdrom; mount oracle cluster agent
    cd /cdrom/cdrom0
    pkgadd -d .
    eject cdrom
    cd /opt/VRTSllt
    cp llttab /etc
    cd /etc
    vi /etc/llttab — uncomment/change following: vi /etc/gabtab cd /etc/rc2.d
    start llt and gab on both machines
    /sbin/lltconfig -a list [check for all 3 interfaces]
    gabconfig -a [check for membership]
    add /sbin and /opt/VRTSvcs/bin to /.profile PATH]

On ONE machine:

      • listener
        oracle
        mount
        volumes
        diskgroup
        vip
        nic
    • update systema and systemb
      update SystemList and AutoStartList
      add diskgroups
      IP qfe1 nic-qfe1
      add mountpoints
      update oracle info
      build dependancies

  • mkdir /etc/VRTSvcs/conf/config
    cd /etc/VRTSvcs/config
    cp *.cf config
    cd sample-oracle
    cp main.cf ../config [may need to copy other files - check]
    cd ../config
    vi main.cf hacf -verify .

manually stop listener [lsnrctl stop]
manually stop db [svrmgrl; connect internal ; shutdown immediate]
take mountpoints out of vfstab
hastart [start cluster]
hagrp -switch oragrp -to systemb [test switchover]

run veritas testing on both machines

Fix Invalid dmpnodename for disk device error

When vxconfigd is not running or is disabled, an error message is received when trying to do a vxdisksetup on a disk.

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c2t5d0s2     sliced    -            -            error

# vxdctl stop
# vxdctl mode
mode: not-running

# vxdisksetup -i c2t5d0
vxdisksetup: c2t5d0s2: Invalid dmpnodename for disk device c2t5d0
Please run vxdctl enable to add new dmpnode

# vxconfigd -m disable
# vxdctl mode
mode: disabled

# vxdisksetup -i c2t5d0
vxdisksetup: c2t5d0s2: Invalid dmpnodename for disk device c2t5d0
Please run vxdctl enable to add new dmpnode

# vxdctl enable
# vxdctl mode
mode: running

# vxdisksetup -i c2t5d0

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c2t5d0s2     sliced    -            -            online

The disk has now been initialized.

Solaris SPARC Boot Sequence

The following represents a summary of the boot process for a Solaris 2.x system on Sparc hardware.

Power On: Depending on the system involved, you may see some output on a serial terminal immediately after power on. This may take the form of a Hardware Power ON message on a large Enterprise server, or a “‘” or “,” in the case of an older Ultra system. These indications will not be present on a monitor connected directly to the server.

POST: If the PROM diag-switch? parameter is set to true, output from the POST (Power On Self Test) will be viewable on a serial terminal. The PROM diag-level parameter determines the extent of the POST tests. (See the Hardware Diagnostics page for more information on these settings.) If a serial terminal is not connected, a prtdiag -v will show the results of the POST once the system has booted. If a keyboard is connected, it will beep and the keyboard lights will flash during POST. If the POST fails, an error indication may be displayed following the failure.

Init System: The “Init System” process can be broken down into several discrete parts:

  • OBP: If diag-switch? is set, an Entering OBP message will be seen on a serial terminal. The MMU (memory management unit) is enabled.
  • NVRAM: If use-nvramrc? is set to true, read the NVRAMRC. This may contain information about boot devices, especially where the boot disk has been encapsulated with VxVM or DiskSuite.
  • Probe All: This includes checking for SCSI or other disk drives and devices.
  • Install Console: At this point, a directly connected monitor and keyboard will become active, or the serial port will become the system console access. If a keyboard is connected to the system, the lights will flash again during this step.
  • Banner: The PROM banner will be displayed. This banner includes a logo, system type, PROM revision level, the ethernet address, and the hostid.
  • Create Device Tree: The hardware device tree will be built. This device tree can be explored using PROM monitor commands at the ok> prompt, or by using prtconf once the system has been booted.

Extended Diagnostics: If diag-switch? and diag-level are set, additional diagnostics will appear on the system console.

auto-boot?: If the auto-boot? PROM parameter is set, the boot process will begin. Otherwise, the system will drop to the ok> PROM monitor prompt, or (if sunmon-compat? and security-mode are set) the > security prompt.

The boot process will use the boot-device and boot-file PROM parameters unless diag-switch? is set. In this case, the boot process will use the diag-device and diag-file.

bootblk: The OBP (Open Boot PROM) program loads the bootblk primary boot program from the boot-device (or diag-device, if diag-switch? is set). If the bootblk is not present or needs to be regenerated, it can be installed by running the installboot command after booting from a CDROM or the network. A copy of the bootblk is available at /usr/platform/`arch -k`/lib/fs/ufs/bootblk

ufsboot: The secondary boot program, /platform/`arch -k`/ufsboot is run. This program loads the kernel core image files. If this file is corrupted or missing, a bootblk: can't find the boot program or similar error message will be returned.

kernel: The kernel is loaded and run. For 32-bit Solaris systems, the relevant files are:

  • /platform/`arch -k`/kernel/unix
  • /kernel/genunix

For 64-bit Solaris systems, the files are:

  • /platform/`arch -k`/kernel/sparcV9/unix
  • /kernel/genunix

As part of the kernel loading process, the kernel banner is displayed to the screen. This includes the kernel version number (including patch level, if appropriate) and the copyright notice.

The kernel initializes itself and begins loading modules, reading the files with the ufsboot program until it has loaded enough modules to mount the root filesystem itself. At that point, ufsboot is unmapped and the kernel uses its own drivers. If the system complains about not being able to write to the root filesystem, it is stuck in this part of the boot process.

The boot -a command singlesteps through this portion of the boot process. This can be a useful diagnostic procedure if the kernel is not loading properly.

/etc/system: The /etc/system file is read by the kernel, and the system parameters are set.

The following types of customization are available in the /etc/system file:

  • moddir: Changes path of kernel modules.
  • forceload: Forces loading of a kernel module.
  • exclude: Excludes a particular kernel module.
  • rootfs: Specify the system type for the root file system. (ufs is the default.)
  • rootdev: Specify the physical device path for root.
  • set: Set the value of a tuneable system parameter.

If the /etc/system file is edited, it is strongly recommended that a copy of the working file be made to a well-known location. In the event that the new /etc/system file renders the system unbootable, it might be possible to bring the system up with a boot -a command that specifies the old file. If this has not been done, the system may need to be booted from CD or network so that the file can be mounted and edited.

kernel initialized: The kernel creates PID 0 ( sched). The sched process is sometimes called the “swapper.”

init: The kernel starts PID 1 (init).

init: The init process reads the /etc/inittab and /etc/default/init and follows the instructions in those files.

Some of the entries in the /etc/inittab are:

  • fs: sysinit (usually /etc/rcS)
  • is: default init level (usually 3, sometimes 2)
  • s#: script associated with a run level (usually /sbin/rc#)

rc scripts: The rc scripts execute the files in the /etc/rc#.d directories. They are run by the /sbin/rc# scripts, each of which corresponds to a run level.

Debugging can often be done on these scripts by adding echo lines to a script to print either a “I got this far” message or to print out the value of a problematic variable.

Add and Configure LUNs in Solaris and Veritas Cluster

ON BOTH NODES
Step 1. Make a copy of the output when you run format

Step 2. Edit /kernel/drv/sd.conf and add an entry for the LUNs you are adding

name="sd" parent="lpfc" target=17 lun=42 hba="lpfc0";

Step 3. Run the following commands so that Solaris can see the new LUNs as disks.

#> update_drv -f sd
#> devfsadm

Step 4. Run format and select the new disks. It should show up as c3t17dxx. Select that disk and label it when asked. Do this for all added LUNs.

Step 5. Run vxdiskadm. Select 1. Add or initialize one or more disks. You can type list now to view all of the disks that Veritas Cluster currently sees. The uninitialized ones will be labeled as such. Type in the c3t17dxx designation to select it and accept all of the default except for the following:

  • When it asks for the disk group, enter the name of the disk group that contains the volume that you wish to increase its size
  • When it asks if you would like to encapsulate the disk, say no. You will then be prompted to initialize, say yes.

Step 6. Make a note of the disk designations that Veritas Cluster provides (ex: mydisk12)

Step 7. Run vxassist to get an idea if the disk initialization really worked as expected. This will allow you to see the maximum amount of space the volumes can be expanded:

#> vxassist -g mydiskgroup maxgrow myvolume mydisk##

It will show you the maximum size that the volume can be expanded in clusters and MB.

Step 8. It is now a good idea to get a copy of the current sizes of the disks. You will need this later. So do a df -th and record the sizes of the volumes you are wanting to expand.

Step 9. Grow the filesystems you need by issuing the following command:

/etc/vx/bin/vxresize -F vxfs -g mydiskgroup myvolume +42g mydisk##

Step 10. Compare the output of df -th to the copy you saved earlier

Extend file system in Veritas and UFS

Extending a File System

If you find that your vxfs file system is not large enough, you can increase the length of the file system. If the underlying volume is not already large enough to accommodate the new file system size, you need to increase the size of the volume first (using vxassist). The new size of the file system is specified in units of 512-byte blocks (or sectors).

Extend a vxfs file system as follows:

fsadm [-F vxfs] [-b newsize] [-r rawdev] mount_point

where newsize is the size to which the file system will grow (in sectors), and rawdev can optionally be used to specify the pathname of the raw device if it cannot be determined by fsadm.

To extend a vxfs file system using /pubs as a mount point, enter:

fsadm -F vxfs -b 22528 /pubs

Shrinking a File System

If you find that you have allocated more space for your vxfs file system than you need, you can decrease the size of the file system.

________________________________________

Note: In cases where data is allocated towards the end of the file system, shrinkage may not be possible and you may have to reorganize the file system. See the VERITAS File System (VxFS) System Administrator’s Guide for details.

________________________________________

Shrink a vxfs file system as follows:

fsadm [-F vxfs] [-b newsize] [-r rawdev] mount_point

where newsize is the size to which the file system will grow (in sectors), and rawdev can optionally be used to specify the pathname of the raw device if it cannot be determined by fsadm.

To shrink a vxfs file system to 20480 sectors using /pubs as a mount point, enter:

fsadm -F vxfs -b 20480 /pubs

With vxresize

Increase a ufs or vxfs file system and volume. Can only decrease vxfs

If the system is a layered volume you cannot add on subdisks to resize

To check space free (in sectors) divide by 2 then 1024 for megs

# vxdg -g datadg free

# vxassist -g &ltdg> maxsize &ltlayout=stripe> &ltncols=x>

# vxassist -g &ltdg> maxsize &ltlayout=stripe> &ltncols=x> &ltvolume>

# vxassist -g &ltdg> maxgrow &ltvolume>

Grow volume to 2 Gb

# vxassist -g &ltdg> growto &ltvolume> 2G

/etc/vx/bin/vxresize [-bsx] [-F fstype] [-g diskgroup] [-t tasktag] volume new_length [medianame...]

# /etc/vx/bin/vxresize -g datadg elsuser +500m

# /etc/vx/bin/vxresize -g datadg db002 -3g

# /etc/vx/bin/vxresize -g datadg redo_archlogs_v +1g

# /etc/vx/bin/vxresize -g datadg redo_archlogs_v 9929612k

# df -k|grep archlogs

/dev/vx/dsk/datadg/redo_archlogs_v 9929612 4304427 5273664 45% /redo_archive_logs

add space to mount point using growfs (if vxresize does not work)

# vxassist -g datadg growby volume 1g

# growfs -M /db003 /dev/vx/rdsk/rootdg/vol3

add space using subdisks and fsadm

Add a subdisk

tardis:root(CDBP)$ vxsd -g datadg assoc export_home_pl A1R3-OUTER-8

Now grow the volume export_home_v :

tardis:root(CDBP)$ vxvol -g datadg set len=4879196 export_home_v

Extend the filesystem : - use vxprint -Ath

v index02_v fsgen ENABLED ACTIVE 4879196 ROUND -

tardis:root(CDBP)$ /opt/VRTSvxfs/sbin/fsadm -b 4879196 /export/home

SSH installation

#########################
# Step 1 # Copy package from /u/goelc/sol_patch/< respective sol volume>
#########################
openssh-4.7p1-sol8-sparc-local.gz
openssl-0.9.8h-sol8-sparc-local.gz
prngd-0.9.25-sol8-sparc-local.gz
zlib-1.2.3-sol8-sparc-local.gz

#########################
# Step 2 # uncompress all the gzip’d files
#########################

gunzip –d openssh-4.7p1-sol8-sparc-local.gz
gunzip –d openssl-0.9.8f-sol8-sparc-local.gz
gunzip –d prngd-0.9.25-sol8-sparc-local.gz
gunzip –d zlib-1.2.3-sol8-sparc-local.gz

#########################
# Step 3 # install the packaes in this sequence
#########################
#pkgadd -d openssl-0.9.8f-sol8-sparc-local

The following packages are available:
1 SMCosslc openssl
(sparc) 0.9.8f

#pkgadd -d prngd-0.9.25-sol8-sparc-local

The following packages are available:
1 SMCprngd prngd
(sparc) 0.9.25

# pkgadd -d zlib-1.2.3-sol8-sparc-local

The following packages are available:
1 SMCzlib zlib
(sparc) 1.2.3

# pkgadd -d openssh-4.7p1-sol8-sparc-local

The following packages are available:
1 SMCosh471 openssh
(sparc) 4.7p1

#########################
# Step 4: Startup Script
#########################
# cp sshd.sh to /etc/init.d/sshd

Make the script executable and create a startup script on run level 2.

# chmod +x /etc/init.d/sshd
# ln -s /etc/init.d/sshd /etc/rc2.d/S99sshd

#########################
# Step 5: same thing for prngd
#########################

# cp prgnd.sh /etc/init.d/prngd
# chmod +x /etc/init.d/prngd
/# ln -s /etc/init.d/prngd /etc/rc2.d/S99prngd

#########################
# Step 6: path it correctly(** remember to add to .profile also)
#########################
export PATH=$PATH:/usr/local/bin:/usr/local/sbin:.

#########################
# Step 7: startup the processes prngd first
#########################
/etc/init.d >./prngd start
starting PRNG daemon
/etc/init.d >Info: Random pool not (yet) seeded
Could not bind socket to /var/spool/prngd/pool: No such file or directory
Jun 1 14:50:19 XXXXXX prngd[2730]: [ID 388259 daemon.alert] Could not bind socket to /var/spool/prngd/pool: No such file or directory

#########################
# Step 8: create the missing folder then try again
#########################
/etc/init.d >mkdir -p /var/spool/prngd
/etc/init.d >./prngd start
starting PRNG daemon
/etc/init.d >Info: Random pool not (yet) seeded

#########################
# Step 9: start ssh process
#########################
/etc/init.d >sshd start
ld.so.1: sshd: fatal: libgcc_s.so.1: open failed: No such file or directory
Killed
/etc/init.d >

# mkdir /var/empty
# chown root:sys /var/empty
# chmod 755 /var/empty
# groupadd sshd
# useradd -g sshd -c ’sshd privsep’ -d /var/empty -s /bin/false sshd

/var/empty should not contain any files.
The default sshd_config file in /usr/local/etc has the last line
Subsystem sftp /usr/libexec/sftp-server
#########################
# Step 10: Create keys for V2
#########################
The errors above are due to the fact that we didn’t create any key pairs for our ssh server.
Create a public key pair to support the new, DSA-based version 2 protocol

/ >/usr/local/bin/ssh-keygen -d -f /usr/local/etc/ssh_host_dsa_key -N “”
Generating public/private dsa key pair.
Your identification has been saved in /usr/local/etc/ssh_host_dsa_key.
Your public key has been saved in /usr/local/etc/ssh_host_dsa_key.pub.
The key fingerprint is:
ce:af:e5:96:e6:94:78:23:93:07:03:3a:0a:d0:90:1f root@XXXXXX
/ >

#########################
# Step 11: Create keys also to cater for V1 (OPTIONAL)
#########################
Create a public key pair to support the old, RSA-based version 1 protocol

/ > /usr/local/bin/ssh-keygen -b 1024 -f /usr/local/etc/ssh_host_rsa_key -t rsa -N “”
Generating public/private rsa key pair.
Your identification has been saved in /usr/local/etc/ssh_host_rsa_key.
Your public key has been saved in /usr/local/etc/ssh_host_rsa_key.pub.
The key fingerprint is:
e9:7f:8d:2c:a1:64:66:4b:87:4a:14:99:0b:69:8e:dd root@XXXXXX
/ >

#########################
# Step 12: startup ssh again
#########################
/usr/local/etc >/etc/init.d/sshd start
starting SSHD daemon
/usr/local/etc >Could not load host key: /usr/local/etc/ssh_host_key
Disabling protocol version 1. Could not load host key
Missing privilege separation directory: /var/empty

#########################
# Step 13: vi sshd_config
#########################
/usr/local/etc >ls
moduli ssh_host_dsa_key ssh_host_rsa_key.pub
prngd ssh_host_dsa_key.pub sshd_config
ssh_config ssh_host_rsa_key
/usr/local/etc >
/usr/local/etc >cat sshd_config | grep -i protocol
#Protocol 2,1

vi to edit ssh_config and uncomment the above line so it will look like this without the ‘#’
Protocol 2,1

/etc/init.d >mkdir -p /var/empty
/etc/init.d >./sshd start
starting SSHD daemon
/etc/init.d >Could not load host key: /usr/local/etc/ssh_host_key
Disabling protocol version 1. Could not load host key

/etc/init.d >ps -ef |grep ssh
root 2937 2816 0 15:31:19 pts/5 0:00 grep ssh
root 2934 1 0 15:31:07 ? 0:00 /usr/local/sbin/sshd

/usr/local/etc >ps -ef| grep prngd
root 2944 2816 0 15:32:37 pts/5 0:00 grep prngd
root 2734 1 0 14:51:10 ? 0:00 /usr/local/sbin/prngd /var/spool/prngd/pool

How to Change root Disk in SVM

  1. Take the backup of / (root) file system and full data backup before doing this activity
  1. Shutdown the services which running over the system
  1. Unmount the file system which all are lying on the storage
  1. Capture the output of # cfgadm –al and see the output

Of c1 controller.

  1. Unconfigure the disk which is having faulty

#cfgadm –c unconfigure c1::dsk/c1t0d0s2

  1. Insert new disk after removing the faulty disk

7. Check the new disk is detected by OS or not

#devfsadm –Cv

#format

  1. Create prtvtoc from source disk to new disk

# prtvtoc /dev/rdsk/c1t1d0s2 > /prtfile

#fmthard – s /prtfile /dev/rdsk/c1t0d0s2

  1. Do a newfs on all the file sytems of the new HDD

and install bootblk on the 0 slice of the new disk.

# newfs < raw device > of all slices of new dis

# installboot bootblk /dev/rdsk/c1t0d0s0

10. # metastat –p and metastat ouputs.

11. Create metadb on the new disk

#metadb –d c1t0d0s7(first delete faulty metadb)

#metadb –afc3 c1t0d0s7

  1. Re-assigning the file system from source to new disk.

metareplace –e <main mirror> <sub mirror>

Example:

metareplace –e d10 c1t0d0s0 (for root)

if it doesn’t work

metaclear all the submirrors on the faulted disk.

Metainit all the slices for submirrro

Metattach the submirror with main mirror.

  1. Install boot in new disk.

# cd /usr/platform/sun4u/lib/fs/ufs

#installboot bootblk /dev/rdsk/c1t0d0s0

  1. Pls wait till the resync of the entire disk and slices completes.
  1. Boot the server from target 0 hard disk

#init 0

ok boot -r

How to Check the Ethernet Link Speed, Status and Duplex in SUN Solaris

Description

When troubleshooting networking issues, it is often helpful to determine the state of an ethernet interface. Solaris offers access to many configurable networking parameters through ndd.

Directions

To determine the ethernet interface link status, duplex, and speed on hme0, run the following commands as superuser:

ndd -set /dev/hme instance 0

ndd -get /dev/hme link_status

ndd -get /dev/hme link_mode

ndd -get /dev/hme link_speed

If you have only one ethernet interface, you can leave out the instance command. Otherwise, you can specify the hme instance number there. The results of the next three commands are either 1 or 0. In each case, the value means:

link_status: 0=down 1=up

link_mode: 0=half duplex 1=full duplex

link_speed: 0=10Mbps 1=100Mbps