Skip to content

Waldemar Mark Duszyk

AIX poznajmy go razem
nie AIX
Lekcje AIXu
Lekcje HACMP
Sutry
Umys
recovering lost mpio disks
Last week something happened to fabrics, switches and God only knows what else. Some hosts
lost some disks but fortunately since we mirror to separate fabrics every volume group withstood
this incident. Later, we had to re-acquire the lost or missing disks which for most host was easily
done executing the cfgmgr command. For a few hosts this operations required few additional
steps that are described here. As the first step, I did rmdev -dl hdisk3 -R but unfortunately
cfgmgr did not bring it back..
# lspv
hdisk0 00cc7261a9f50481 rootvg active
hdisk1 00cc72617b870835 rootvg active
hdisk2 00cc7261e4fdc7b7 entoras1_vg active
hdisk4 00cc7261d207431a mksysbvg active
After a repetitious execution, cfgmgr still cannot deliver one missing disks that is required to
mirror entoras1_vg. Just for kicks, I execute the sanscan utility.
# sanscan
sanscan v2.2
Copyright (C) 2010 IBM Corp., All Rights Reserved

Processing FC device:
Adapter driver: fcs0
Protocol driver: fscsi0
Connection type: fabric
Local SCSI ID: 0x083e00
Local WWPN: 0x10000000c96ef366
Local WWNN: 0x20000000c96ef366
Initializing device information...
Scanning SAN...
SCSI ID LUN ID WWPN WWNN Vendor ID Product ID Rev NACA Qualifier Device Type
Error(s)
--------------------------------------------------------------------------------------------
---------------------------
080c00 0000000000000000 5005076801402afd 5005076801002afd IBM 2145 0000 yes Connected
Disk [hdisk2, path_id 0]
080c00 0001000000000000 5005076801402afd 5005076801002afd IBM 2145 0000 yes Connected
Disk [hdisk4, path_id 0]
081c00 0000000000000000 5005076801402ba4 5005076801002ba4 IBM 2145 0000 yes Connected
Disk [hdisk2, path_id 1]
Page 1 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
081c00 0001000000000000 5005076801402ba4 5005076801002ba4 IBM 2145 0000 yes Connected
Disk [hdisk4, path_id 1]
603d00 0000000000000000 5005076801202a94 5005076801002a94 IBM 2145 0000 yes Connected
Disk [no ODM match]
601d00 0000000000000000 5005076801202d7f 5005076801002d7f IBM 2145 0000 yes
Connected Disk [no ODM match]
4 targets and 6 LUNs found in 0.015390 seconds
Yes, sanscan sees something but cfgmgr is not able to get it. In situation like this one, it is
often helpful to remove the parent (physical adapter) and all its children. Before any removal, we
verify that there is at least one more additional available FC adapter (lsdev -Cc adapter |
grep FC) and that all disks have paths leading through it.
# lspath
Enabled hdisk0 scsi1
Enabled hdisk1 scsi1
Enabled hdisk2 fscsi0
Enabled hdisk4 fscsi0
Enabled hdisk2 fscsi0
Enabled hdisk4 fscsi0
Enabled hdisk2 fscsi1
Enabled hdisk4 fscsi1
Enabled hdisk2 fscsi1
Enabled hdisk4 fscsi1
Knowing that the host has paths to each disk through both FC adapter allows us to proceed. We
start with smitty devices and select the Disable a FC SCSI Protocol Device menu. From
the list, select the first adapter (fcs0) and validate the selection with the RETURN key. Next, get out
of smitty and proceed with removal of this adapter.
# rmdev -dl fcs0 -R
fcnet0 deleted
fscsi0 deleted
fcs0 deleted
Execution of cfgmgr followed with lspv shows a new disk. Lets repeat the same process as
before but this time we start disabling fscs1 and next removing it. With the new disk back on
board, we have to verify this it is the missing mirror that we will use to re-mirror entoras1_vg.
What will determine that this is the right disks? First, it must have the same size as hdisk2.
# bootinfo -s hdisk2
599040
# bootinfo -s hdisk3
599040
Next, both disks have to belong to different fabrics/controllers.
# lsattr -El hdisk2 | awk '/unique_id/ {print $2}'
33213600507680181003148000000000002ED04214503IBMfcp
# lsattr -El hdisk3 | awk '/unique_id/ {print $2}'
332136005076801830037C00000000000037904214503IBMfcp
Page 2 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
Indeed, hdisk2 belongs to SVC 3148 and hdisk3 to SVC 037C two different fabrics, we
can proceed with mirroring.
# extendvg -f entoras1_vg hdisk3
# mirrorvg -S -c 2 entoras1_vg hdisk3
PS
Ramon, thanks for showing me how to use awk pattern matching .
Posted in Real life AIX.
Tagged with disk, extendvg, fcs, fscsi, mirrorvg, missing, rmdev.
No comments
By MarkD:-) 5 March 2012
0 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.
Leave a Reply

Some HTML is OK
Name (required)
Email (required, but never shared)
Web
Post comment or, reply to this post via trackback.
mirroring progress in aix identifying HDS ShadowImage disks in mpio environment
Subscribe

About Waldemar Mark Duszyk
A quote from a reader: Everything can be explain in a simple way, even rocket science. And you
proved it.
Page 3 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
Share |
Search for: Search
Popular Tags
Najnowsze Posty
fixing offline paths in dlnkmgr environment.
matching Hitach and AIX FC adapters
san design quide for aix with mpio, sdddrv, pcmdrv
identifying HDS ShadowImage disks in mpio environment
recovering lost mpio disks
mirroring progress in aix
mkuser does not work..
HDS (USP_V) hdisk/controller relationship
local print queue removal in AIX
installing Shadow Image on AIX host
removing orphaned home directories
using NIM to install mksysb image building spot from mksysb
mounting ISO images in AIX6.1
in-place editing of files
FC Adapters and the associated with them devices (disks and tapes)
Comments
MarkD:-) on mount WINdows shares on AIX host without SAMBA
John S on mount WINdows shares on AIX host without SAMBA
MarkD:-) on Wprowadzenie do HACMP
Brian weber on Wprowadzenie do HACMP
Mike Forster on vi fails to open large files
MarkD:-) on Rygorystyczna kontra Super Rygorystryczna polisa topograficzna
lukas on Rygorystyczna kontra Super Rygorystryczna polisa topograficzna
slashbone on mj lpar zawsze koczy boot w SMS
morroccan on Lekcja 4: Pracujc z dyskami
Sebastian on mirroring progress in aix
Mark on mirroring progress in aix
Ramon on mirroring progress in aix
MarkD:-) on local print queue removal in AIX
Ramon on local print queue removal in AIX
Marco on Lekcja 4: Pracujc z dyskami
Page 4 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
AIX Links
A nice AIX site from Toronto, Canada Check its Samba pages.
Aix 6.1 + PowerVM + HDLM 3.01 + GPFS + Oracle Clusterware + Oracle RAC 11GR2
Good AIX info with links

AIX Installation Methods

Anthony English, AIX blog An excelent source of AIX info from Australia
Aussie Storage Blog with Anthony Vanderwerdt An excellent source of XIV info
Chris Gibson, AIX Blog An excelent source of AIX info from Australia
Everything AIX in one PLACE
Everything XIV
HMC Documentation
IBM Power Systems Hardware Information Center
IBM White Papers
Oracle Tunning Kevin Closson Blog
PERL little gems
Sed An Introduction and Tutorial by Bruce Barnett
SED little gems
Stefan Radtke XIV Storage Blog
StorageBuddhist SAN advice from New Zealand
Storm M Harris The starter Guide to NIM
Unix Shell Scripts Tips
William Favorite All Things New
Page 5 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
Live Traffic Feed
See your visitors
in RealTime! Get the
Free Live Traffic Feed
Get Feedjit Now!
A visitor from
Nottingham viewed "fsck
all file systems of a
volume group
Waldemar Mark
Duszyk" 1 min ago
A visitor from Oradea,
Bihor viewed "su does
not work Waldemar
Mark Duszyk" 2 mins
ago
A visitor from Denver,
Colorado viewed "Pass
Through Authentication
between TDS Ldap
servers and
ActiveDirectory over
SSL Waldemar Mark
Duszyk" 6 mins ago
A visitor from Madras,
Tamil Nadu viewed
"Waldemar Mark
Duszyk - [lang_en]AIX -
learn it and share it
[/lang_en][lang_pl]AIX -
poznajmy go razem
[/lang_pl]" 7 mins ago
A visitor from Gosport,
Hampshire viewed
"changing passwords
remotely via ssh script
Waldemar Mark
Duszyk" 7 mins ago
A visitor from Paris, Ile-
de-France viewed "0516-
1339 mkvg: Physical
volume contains some
3rd party volume group
Waldemar Mark
Duszyk" 12 mins ago
A visitor from
Yekaterinburg,
Sverdlovsk viewed
"Working with HMC
command line
Page 6 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
free hit counter
Language

Kalendarz Postw
M T W T F S S
Feb
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31
March 2012
Consolidate Your
Servers
Experience Why
VMware is The
Industry's
Virtualization Choice.
VMware.com/vSphere_SMB
Linux for the Real
World
Practical Linux training
User directed,so get
your input in.
cbtnuggets.com/practical/lin
Oracle Platinum
Partner
Specialists in Oracle
database, middleware
& hardware solutions
www.esteem.co.uk/oracle
Shared Storage for
OS X
Turn your Mac into a
server and share mxf
media and avp
projects!
www.xmcorebeta.wordpress
4th Cloud
Computing Forum
Learn all about
Virtualization - Over
150 Free to Attend
Seminars
www.cloudwf.com/London
Page 7 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
Password Manager
Securely store admin
passwords Ensure
internal control,
compliance
ManageEngine.com/Passwor
iSCSI SAN & NAS
Cost effective iSCSI
SAN solutions for
VMware, Linux &
Windows.
www.dnuk.com
IBM Launch
On 11th April IBM will
introduce a new
category of computing.
See how!
ibm.com/ExpertIntegratedS
Next Gen SAN
Technology
150K IOPS, 100Tb
SAN for 35K from
The UK's leading
GreenBytes Partner
www.enablesit.co.uk
Nexenta ZFS
Appliances
Enterprise Class, high
speed NAS/iSCSI/FC
VDI storage systems
www.nas.uk.com
Page 8 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
Archives
March 2012 (8)
February 2012 (8)
January 2012 (10)
December 2011 (5)
November 2011 (10)
October 2011 (7)
September 2011 (4)
August 2011 (10)
July 2011 (6)
June 2011 (2)
May 2011 (3)
April 2011 (7)
March 2011 (11)
February 2011 (12)
January 2011 (7)
December 2010 (10)
November 2010 (3)
Upgrade to
Windows 7
Discover how Windows
7 & Office 2010 can
benefit your business
microsoft.com/XPUpgrade
Download Google
Chrome
A free browser that
lets you do more
online. Download
instantly!
www.google.com/chrome
Pro and Network
Storage
Backup, data safety,
fibre storage for audio
and video industries.
www.sonnettech.com
Sun Microsystems
Support
System Maintenance
and Support Flexible
service plans for Sun
www.Criticalimpact.co.uk
Fix Windows
Errors
Scan your PC and be
error free. Computer
repair software. Fix
now!
www.iolo.com/download
Page 9 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl
October 2010 (9)
September 2010 (12)
August 2010 (9)
July 2010 (4)
June 2010 (7)
May 2010 (8)
April 2010 (4)
March 2010 (6)
February 2010 (1)
January 2010 (11)
December 2009 (6)
November 2009 (7)
October 2009 (6)
September 2009 (4)
August 2009 (6)
July 2009 (6)
June 2009 (9)
May 2009 (13)
April 2009 (7)
March 2009 (8)
February 2009 (7)
December 2008 (1)
November 2008 (1)
Proudly powered by WordPress and Carrington.
2008-2012 www.wmduszyk.com - best viewed with your eyes.
Page 10 of 10 recovering lost mpio disks Waldemar Mark Duszyk
22/03/2012 http://www.wmduszyk.com/?p=8156&langswitch_lang=pl