<Insert Picture Here>

Thumper & Thor Technical Product Training Presentation

TOI
Greg Huff Sun Microsystems, Inc.

Base Content Slide

First-level bullet
Second-level bullet Third-level bullet Fourth-level bullet Fifth-level bullet

Roll Call Ask questions as we go. Interrupt me if necessary. May need your help. Remind to take one short break.

Thumper & Thor TOI Notes

Names used in this TOI x4500 Thumper x4540 Thor Ask questions as we go along.

x4540 Thor 2-Socket Enterprise Server with 48 SATA hard drives direct attached

Compute
2 AMD Opteron 2000 Series CPUs, Quad-Core 8-way performance in a 4 RU form factor, with lots of disks 16 DIMM slots, 8 per CPU

Storage, lots and lots of storage (and you get a free 8-way server)

I/O 3x PCI-e 8-lane slots 4x Gigabit Ethernet ports 48x SATA 3.5 disk drives Availability N+1 redundant hot-swappable power supplies and fans Software RAID Management and OS
IPMI 2.0; remote KVMS with dedicated 10/100 Ethernet port Solaris OS (pre-installed) Linux and Windows OS post-GA

x4500 Thumper 2-Socket Enterprise Server with 48 SATA hard drives direct attached

Compute
2 AMD Opteron 200 Series CPUs, Dual-Core 4-way performance in a 4 RU form factor, with lots of disks 8 DIMM slots, 4 per CPU

Storage, lots and lots of storage (and you get a free 4-way server)

I/O 2x PCI-X slots 4x Gigabit Ethernet ports 48x SATA 3.5 disk drives Availability N+1 redundant hot-swappable power supplies and fans Software RAID Management and OS
IPMI 2.0; remote KVMS with dedicated 10/100 Ethernet port Solaris OS (pre-installed) Linux and Windows OS post-GA

<Insert Picture Here>

Features

Thumper & Thor Front Features

48 Hot Swappable SATA HDDs Redundant, Hot Swappable Fans Locate and Power Buttons, Service Action and Power OK LED's

2 USB 2.0 Ports

Thor vs Thumper Key Differences

F e a tu re Thum per (x 4 5 0 0 ) AM D 8132 and AM D 8111 In te l 8 2 5 4 6 G B Thor (x 4 5 4 0 ) N v id ia M C P 5 5 a n d IO 5 5 B u ilt -in n V id ia c o n t r o lle r (M C P 5 5 /IO 5 5 ) w ith M a r v e ll 88E1121R dual PH Y LSI SAS1068E M PT R o w o r ie n te d , s ta r tin g fr o m th e f r o n t r o w , w it h d r iv e s 0 - 1 1 a c r o s s th e fro n t fro m le ft to rig h t F r o m le ft to r ig h t: 3 -2 -5 -4 -1 -0 2 : d r iv e s 0 & 1 (P h y s 0 & 4 o n c o n tr o lle r 3 ) C o lu m n o r ie n t e d , s t a r tin g fr o m th e le ft-m o s t c o lu m n , w ith d r iv e s 0 -3 u p t h e le ft s id e f r o m f r o n t to back F r o m le ft to r ig h t: 0 -1 -2 -3 -4 -5 4 : d r iv e s 0 & 1 (P h y s 0 & 1 o n c o n tr o lle r 0 ) p lu s d r iv e s 8 & 9 (P h y s 0 & 1 o n c o n tr o lle r 1 ) 1 1 0 o p t io n a l w it h 3 r d P S U P C I- e

C h ip s e t G ig a b it E th e r n e t C o n t r o lle r

H a r d D is k C o n t r o lle r D r iv e C o n t r o lle r D r iv e r H a r d D is k M a p p in g

M a rv e ll 8 8 E 6 0 8 1

C o n tr o lle r M a p p in g B o o t d r iv e s

Pow er PCI

2 2 0 V o n ly P C I-x

Service Processor

GRASP

Embedded on MB

Thor CPU Block Diagram

D D R D C /D C O p te r o n V R M
H T

D D R D C /D C O p te r o n V R M

D D R 2 D IM M x 8

M e z z a n in e C o n n e c t o r

M e z z a n in e C o n n e c t o r

D D R 2

A M D S le d g e H am m er 0

0
1

H T

A M D S le d g e 0 H am m er

H T

D D R 2

2
H T

H T

I2 C

I2 C ID PRO M I2 C S ys M on/ F a n C o n tr o l

D D R 2 D IM M x 8

Thor HT Links Configuration

CPU0 link 0 connects to MCP55 CPU0 link 1 connects to nothing, it is unused CPU0 link 2 connects to CPU1 link 1 CPU1 link 0 connects to IO55A CPU1 link 1 connects to CPU0 link 2 CPU1 link 2 connects to IO55B

Thumper HT Links Configuration

CPU0 Link 0 => 8132 (marvell controllers bus5/func1 and bus6/func1) / 8132 (1GbEx4, bus7 and bus8) CPU0 Link 1 => 8132 (marvell controllers bus1/func1 and bus2/func1) / 8111 (Southbridge, bus0 and bus3) CPU0 Link 2 => CPU1 Link 1 CPU1 Link 0 => 8132 (PCI-X Slot 0 busd and Slot 1 buse) CPU1 Link 1 => CPU0 Link 2 CPU1 Link 2 => 8132 (marvell controllers busa/func1 and busb/func1)

Thor IO Block Diagram

LS I S A S10 68e P C I- e S lo t 0 - x 8 P C I- e S lo t 1 - x 8 P C I- e S lo t 2 - x 8
A ll d is k c o n t r o lle r in t e r f a c e s a r e P C I-e x 8 P C I-e P C I-e H T P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6

P C I-e

n V i d ia I O 5 5

P5

LSI SA S1 068 e LS I S AS 106 8e

P1

C PU0P0 M e z z a n in e C o n n e c t o r

M e z z a n in e C o n n e c t o r

Q uad R J45

P o rt 2 w it h G ig E M a g n e t ic s P o rt 3

M 88E 1121R
D u a l G ig E P H Y

C PU 1P 2

H T

2 0

3 1

D u a l G ig E P H Y

H T I2 C

LS I SA S1 068e

P o rt 1

R G M II

C o m p a c t F la s h S lo t

I2 C

Q S M ux LSI S AS 106 8e ID E
L P C

L P C

Q S C P LD
X C 9572X L S P I2 L P C

P1

U S B x2

n V id ia M C P 5 5

I2 C

A T97S C 3203

M 50LP W 080

P0

TP M U S B x2 R J45

B IO S

P O R T 7 P O R T 6 P O R T 5 P O R T 4 P O R T 3 P O R T 2 P O R T 1 P O R T 0 P P P P P P P P O O O O O O O O R R R R R R R R T T T T T T T T 7 6 5 4 3 2 1 0

H D D 15 H D D 14 H D D 13 H D D 12 H D D 11 H D D 10 H D D 9 H D D 8 H H H H H H H H D D D D D D D D D D D D D D D D 7 6 5 4 3 2 1 0

P5

S P I S e ria l X C V R M ux

P C I 3 2 b it 33M H z

AST2000
IR Q A ,B ,C ,D ID S E L = A D 1 9

I2 C
U S B x1

ID P R O M

S ys M on/ F a n C o n tr o l

Q S

LS I S A S10 68e

U S B x2

C o n n e c to r

P o rt 0

M 88E 1121R

CP U1P 0

R G M II

P0

P0 n V i d ia I O 5 5 P1 P5

Thumper Block Diagram

P o w e r S u p p ly P o w e r S u p p ly P o w e r S u p p ly /B a tte r y B a c k u p U n it T h u m p e r P ro c e s s o r (C P U ) B o a rd - H e a v e n ly
H T H T

C o n tr o l a n d S ta tu s 12V P o w e r B a c k p la n e - B o r e a l C o n tr o l a n d S ta tu s 12V C o n tr o l a n d S ta tu s 12V

A M D O p te r o n

DDR

H T

M e z z a n in e C o n n e c to r
H T

M e z z a n in e C o n n e c to r

H T

DDR

M e z z a n in e C o n n e c to r

M e z z a n in e C o n n e c to r

A M D O p te r o n

T h u m p e r IO C o n tr o lle r B o a r d - S u g a r b o w l A M D 81 32 H T - > P C I-X B rid g e P C I- X S lo t 0 - 1 3 3 M H z P C I- X S lo t 1 - 1 3 3 M H z 1 0 /1 0 0 / 1000 1 0 /1 0 0 / 1000 1 0 /1 0 0 / 1000 1 0 /1 0 0 / 1000

P C I-X H T

C o n tr o l a n d S ta tu s 12V 3 .3 V & 5 .0 V
8 -p o rt S -A T A C o n t r o lle r 8 -p o rt S -A T A C o n t r o lle r 8 -p o rt S -A T A C o n t r o lle r 8 -p o rt S -A T A C o n t r o lle r 8 -p o rt S -A T A C o n t r o lle r 8 -p o rt S -A T A C o n t r o lle r

C n tl /S ta t 12V

P C I-X

P C I- X

H T

P C I-X

S -A T A x 8

H T

C F s lo t N O T S U P P O R TE D
LP C

H T

ID E A M D 8111 S o u th B rid g e

H T

H T

AM D 813 2 H T - > P C I- X B r id g e

D ual 1 G ig E M A C /P H Y

P C I- X

M e z z a n in e C o n n e c to r

M e z z a n in e C o n n e c to r

P C I-X

S -A T A x 8

P C I-X

S -A T A x 8

D is k B a c k p la n e - M a m m o th

D ual 1 G ig E M A C /P H Y

M e z z a n in e C o n n e c to r

M e z z a n in e C o n n e c to r

A M D 81 32 H T - > P C I-X B rid g e

P C I-X

S -A T A x 8

S e r ia l

Q S

TP M

Q S U S B x1

B IO S

C n tl /S ta t 12V
F le x C ir c u it

A M D 8132 H T - > P C I- X B ri d g e

P C I-X

S -A T A x 8

U S B x2 G F X /S P B o a r d D ia m o n d P e a k

U S B x2 US B x2

PC I

U S B x1

U S B H ub

I n d ic a t o r B o a rd N o r th s ta r

U S B 2 .0 C o n tr o l le r

P C I-X

S -A T A x 8

F a n T ra y H o m e w o o d

F a n T ra y H o m e w o od

D s s D is ki D ki D ki s k s D is k s D i D kis k D D D i D ki s k D is kis kis k D s s D is ki D ki D ki s k s D is k s D i D kis k D D D i D ki s k D is kis kis k s D is k D D D i D ki s k D is kis kis k D D s D is k D kis kis ki D ki s k D is D s s D is ki D ki D ki s k D i D kis kis k s D D D D D isk D is kis kis ki s k

A M D 8 132 H T -> P C I- X B ri d g e

Thor CRU/FRU List PROPOSED, SUBJECT TO CHANGE

System Boards Power Backplane Indicator Board (inc. cable)

FRU FRU

System Modules Super FRU FRU - includes disk bp & sheetmetal Controller Assembly FRU - includes CPU and IO broads

CPU's AMD Opteron Processors Hard Drives HDD 3.5 SAS Power Power Supply Memory Memory Modules Fans Fan Modules

FRU CRU CRU CRU CRU

Post RR CRU/FRU list is here: http://panacea.central.sun.com/twiki/pub/Products/ProdKnowledgeSunFireX4540/CruFruList2.txt

Thumper CRU/FRU List PROPOSED, SUBJECT TO CHANGE

System Boards Power Backplane FRU Indicator Board (inc. cable) FRU Graphics/SP module CRU System Modules Super FRU FRU - includes disk bp & sheetmetal Controller Assembly FRU - includes CPU and IO broads

CPU's AMD Opteron Processors Hard Drives HDD 3.5 SAS Power Power Supply Memory Memory Modules Fans Fan Modules

FRU CRU CRU CRU CRU

Post RR CRU/FRU list is here: http://panacea.central.sun.com/twiki/pub/Products/ProdKnowledgeSunFireX4540/CruFruList2.txt

<Insert Picture Here>

Components

Thor Opteron Rev F CPU Board

CPU 1

CPU0 8 DDR2 Memory Modules

Battery Fault LED

CPU Fault LEDs

Fault Remind Button

Thor CPU's

AMD Opteron
AMD Opteron 2000 Series
Quad-Core 2-socket configuration only Rev F No mixing of speeds or steppings AMD PowerNow! (disabled by default)

Thor CPU Module LED Behavior LEDs identify failed CPU or memory components on CPU board Press and hold the blue fault remind button On board capacitor maintains power to the board for a few minutes after removal from the system DIM
M Faul t LED s

Batt Faul t LED

Cap

CPU DIM Faul M t Fault LEDFault LED s Remind s Button

Thumper Opteron Rev E CPU Board

Fault Remind Button Battery Fault LED CPU Fault LEDs VRM 4 DDR ECC Memory Modules

CP U

Thumper CPU's

AMD Opteron
AMD Opteron 200 Series
Dual-Core 2-socket configuration only Rev E No mixing of speeds or steppings AMD PowerNow! (disabled by default)

Thumper CPU Module LED's identify failed CPU LED Behavior or memory components on CPU board Fault Batt Press and hold the blue Remind Faul fault remind button Button t On board capacitor LED maintains power to the s board for a few minutes after removal from the system

CPU Faul t LED s

Capacitor

Individua l DIMM Fault LED's

Thor System Memory

8 DIMM slots per CPU 128-bit DDR2 interface (2x64-bit data+2x8-bit ECC) ECC, Registered DDR667 SDRAM 1 2 GB DIMM modules 4 GB DIMM's Q?CY?? Up to 64 GB memory Low profile 3.05mm ECC Chipkill enabled in BIOS by default Interleaving by default, Bank Auto, Node Disabled

Thor Memory Placement LED fault indicators help identify faulty DIMM's Populated by pair, white pair first
White pair = DIMM6 + DIMM7 White pair = DIMM2 + DIMM3 Black pair = DIMM4 + DIMM5 Black pair = DIMM0 + DIMM1
Sun Fire X4540 Supported DIMM Part Numbers Component Part Numbers: 2-GB DIMMs, pair (2 DIMMs, 4 GB total), 1903 registered ECC memory, 8 slots/system X4231A-Z (X-Option)

#541-

Thor Memory Configuration

Model/Family Independent 8 DIMMs/CPU Populate from outside -> in Matched Pairs Use DIMMs from NSG memory Qual matrix spreadsheet Barcelona Specifics 2/4 DIMMs 800 6/8 DIMMs 667

Thor Memory Placement

Thumper Memory Placement LED fault indicators help identify faulty DIMM's Populated by pair, white pair first
White pair = DIMM0 + DIMM1 Black pair = DIMM2 + DIMM3

Thor Chipsets

nVidia MCP55/IO55
nVidia MCP-55
1GHz 8GB/sec HyperTransport 3 x PCIe 8-lane ports 7 x USB 2.0 ports 2 x GigE ports Legacy PCI (to AST2000) LPC to BIOS flash and TPM IDE to Compact Flash slot

nVidia I0-55
1GHz 8GB/sec HyperTransport 3 x PCIe 8-lane ports 2 x GigE ports

Thumper Chipsets

AMD 8111 and 8132

AMD 8111 Southbridge AMD 8132 PCI-X Bridge x5 200 MHz HT link to AMD 8132 LPC connection to Super I/O and BIOS Flash PCI connection to ATI Rage XL and USB

Thor SATA Controller LSI SAS1068e

8 SATA ports
Even though the 1068e supports SAS, we will ONLY be using SATA in Thor

Each port capable of 3.0 Gbit/s SATA link Total available SATA bandwidth: 24 Gbit/s
SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device
Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx

LSI SAS1068e

PCI-e Interface (x8)

MCP55

Thumper SATA Controller Marvell 88SX6081

8 SATA ports Each port capable of 3.0 Gbit/s SATA link Total available SATA bandwidth: 24 SATA Device Gbit/s SATA Device
Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx Tx, Rx

SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device SATA Device

Marvell 88SX6081

PCI-X Interface

8132

Thor 6 SATA Controllers on IO Board

LSI SATA controller x 6 Each controller supports 8 HDDs

Thumper 6 SATA Controllers on IO Board Marvell SATA controller x 6 Each controller supports 8 HDDs

<Insert Picture Here>

Disk Details

Thumper and Thor Disk Drives Total of 48 HDD

3.5Serial ATA (SATA) HDD Support SATA II, 3.0Gb/s +12V +5V Power Supply
Hot swappable High performance 7200 RPM High density 250/500/750GB/1TB Vendor: Hitachi, Seagate

Thumper 48 Hard Drive Layout

Boot Disks

Thor 48 Hard Drive Layout

Boot Disks

Boot disks and layout

Thumper Thor

USB CD/DVD = c4 so these change c6 c5 c8 c7 ------>>

ILOM 2.0.2.5 can change:

c3 c2 c5 c4

ILOM 2.0.2.5 workaround

Manually map CD-ROM and Floppy in the JavaRConsole

Thumper: Marvell Error disk mapping

Sep 24 13:33:51 marvell88sx: [ID 812950 kern.warning] WARNING: marvell88sx3: error on port 7: Sep 24 13:33:51 marvell88sx: [ID 517869 kern.info] SError interrupt Sep 24 13:33:51 marvell88sx: [ID 131198 kern.info] SErrors: Sep 24 13:33:51 marvell88sx: [ID 517869 kern.info] Recovered communication error Sep 24 13:33:51 marvell88sx: [ID 517869 kern.info] PHY ready change Sep 24 13:33:51 marvell88sx: [ID 517869 kern.info] 10-bit to 8-bit decode error Sep 24 13:33:51 marvell88sx: [ID 517869 kern.info] Disparity error marvell88sxX: error on port Y = sataX/Y = Target Yon Controller X marvell88sx3: error on port 7 = sata3/7 = Target 7on Controller 3

Common non fatal errors

Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] ASC: 0x0 (no additional sense info), ASCQ: 0x0, FRU: 0x0 1 15:51:59 sawfish-va sata: [ID 801593 kern.notice] NOTICE: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1: 1 15:51:59 sawfish-va port 0: device reset 1 15:51:59 sawfish-va sata: [ID 801593 kern.notice] NOTICE: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1: 1 15:51:59 sawfish-va port 0: link lost 1 15:51:59 sawfish-va sata: [ID 801593 kern.notice] NOTICE: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1: 1 15:51:59 sawfish-va port 0: link established 1 15:51:59 sawfish-va marvell88sx: [ID 812950 kern.warning] WARNING: marvell88sx2: error on port 0: 1 15:51:59 sawfish-va marvell88sx: [ID 517869 kern.info] device disconnected 1 15:51:59 sawfish-va marvell88sx: [ID 517869 kern.info] device connected 1 15:51:59 sawfish-va scsi: [ID 107833 kern.warning] WARNING: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1/disk@0,0 (sd6): 1 15:51:59 sawfish-va Error for Command: read(10) Error Level: Retryable 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Requested Block: 37562914 Error Block: 37562914 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Vendor: ATA Serial Number: 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Sense Key: No Additional Sense 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] ASC: 0x0 (no additional sense info), ASCQ: 0x0, FRU: 0x0 1 15:51:59 sawfish-va scsi: [ID 107833 kern.warning] WARNING: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1/disk@0,0 (sd6): 1 15:51:59 sawfish-va Error for Command: read(10) Error Level: Retryable 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Requested Block: 37563170 Error Block: 37563170 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Vendor: ATA Serial Number: 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Sense Key: No Additional Sense 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] ASC: 0x0 (no additional sense info), ASCQ: 0x0, FRU: 0x0 1 15:51:59 sawfish-va scsi: [ID 107833 kern.warning] WARNING: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1/disk@0,0 (sd6): 1 15:51:59 sawfish-va Error for Command: read(10) Error Level: Retryable 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Requested Block: 37563426 Error Block: 37563426 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Vendor: ATA Serial Number: 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] Sense Key: No Additional Sense 1 15:51:59 sawfish-va scsi: [ID 107833 kern.notice] ASC: 0x0 (no additional sense info), ASCQ: 0x0, FRU: 0x0 1 15:51:59 sawfish-va scsi: [ID 107833 kern.warning] WARNING: /pci@1,0/pci1022,7458@3/pci11ab,11ab@1/disk@0,0 (sd6): 1 15:51:59 sawfish-va Error for Command: read(10) Error Level: Retryable

hdtool aka hd utility Used to identify disks Used to view sd names Used to view Controller path Used to view HDD SMART data Preinstalled at the factory On the Tools and Drivers CD Solaris, Linux, and Windows Solaris version is full featured. Online (and man page) at http://td02.sfbay/ Linked from Thumper Troubleshooting page

Thor (x4540) hd output

# hd ---------------------SunFireX4540------Rear---------------------------3: 7: 11: 15: 19: 23: 27: 31: 35: 39: 43: 47: c0t3 c0t7 c1t3 c1t7 c2t3 c2t7 c3t3 c3t7 c4t3 c4t7 c5t3 c5t7 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 2: 6: 10: 14: 18: 22: 26: 30: 34: 38: 42: 46: c0t2 c0t6 c1t2 c1t6 c2t2 c2t6 c3t2 c3t6 c4t2 c4t6 c5t2 c5t6 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 1: 5: 9: 13: 17: 21: 25: 29: 33: 37: 41: 45: c0t1 c0t5 c1t1 c1t5 c2t1 c2t5 c3t1 c3t5 c4t1 c4t5 c5t1 c5t5 ^b+ ^++ ^b+ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 0: 4: 8: 12: 16: 20: 24: 28: 32: 36: 40: 44: c0t0 c0t4 c1t0 c1t4 c2t0 c2t4 c3t0 c3t4 c4t0 c4t4 c5t0 c5t4 ^b+ ^++ ^b+ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ -------*-----------*-SunFireX4540---*---Front----*---------*--------

Thumper (x4500) hd output # hd

---------------------SunFireX4500------Rear---------------------------36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: c5t3 c5t7 c4t3 c4t7 c7t3 c7t7 c6t3 c6t7 c1t3 c1t7 c0t3 c0t7 ^-^-^-^-^-^-^-^-^-^-^-^-24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: c5t2 c5t6 c4t2 c4t6 c7t2 c7t6 c6t2 c6t6 c1t2 c1t6 c0t2 c0t6 ^-^-^-^-^-^-^-^-^-^-^-^-12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: c5t1 c5t5 c4t1 c4t5 c7t1 c7t5 c6t1 c6t5 c1t1 c1t5 c0t1 c0t5 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: c5t0 c5t4 c4t0 c4t4 c7t0 c7t4 c6t0 c6t4 c1t0 c1t4 c0t0 c0t4 ^b+ ^b+ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ -------*-----------*-SunFireX4500--*---Front-----*-----------*----------

Useful hd options
# hd -h Usage: hd [ -c(olor mode) ] [ -s(ummary) ] [ -p(latform) ] [ -b(ypass) to print SunFireX4500 map ] [ -d(iagnose)] [-f { syslog_file } ] [ -m { adjacent | cross | front2back | diagonal } Mapping pairs ] [ -w { <pci_disk_device_path> } ] [ -a (fdisk pArtition type) ] [ -q (list drive slot number in seQuential list) ] [ -g (list drive slot number in seQuential list with temperature ) ] [ -l (List SunFireX4500/X4540 available disk in physical orders) ] [ -r (List SMART data for all disks in drive slot number) ] [ -R (List SMART data's indivdual id in landscape view for all disks) ] [ -e <cXtY> (List SMART data for specified disk) ] [ -E <cXtY> (List raw hex SMART data for specified disk) ] [ -j (List SunFireX4500/X4540 HBA controller numbers and pci nodes) ] [ -T (List vtoc for all drives for SunFireX4500/X4540 platform) ] [ -t (List vtoc for specified drives) ] [ -i (List cXtY, sd# and PCI path) ] [ -o (List LSI HBA#, Drive Target# and cXtY) ] [ -x (Generate hd_map.html) ]

hd -i
# hd -i c0t0d0 sd24 /pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0 c1t0d0 sd40 /pci@0,0/pci10de,375@b/pci1000,1000@0/sd@0,0 c0t3d0 sd27 /pci@0,0/pci10de,377@a/pci1000,1000@0/sd@3,0 c0t7d0 sd31 /pci@0,0/pci10de,377@a/pci1000,1000@0/sd@7,0 c1t3d0 sd43 /pci@0,0/pci10de,375@b/pci1000,1000@0/sd@3,0 c1t7d0 sd47 /pci@0,0/pci10de,375@b/pci1000,1000@0/sd@7,0 c2t3d0 sd3 /pci@0,0/pci10de,376@f/pci1000,1000@0/sd@3,0 c2t7d0 sd7 /pci@0,0/pci10de,376@f/pci1000,1000@0/sd@7,0 c3t3d0 sd35 /pci@3c,0/pci10de,377@a/pci1000,1000@0/sd@3,0 c3t7d0 sd39 /pci@3c,0/pci10de,377@a/pci1000,1000@0/sd@7,0 c4t3d0 sd19 /pci@3c,0/pci10de,375@b/pci1000,1000@0/sd@3,0 c4t7d0 sd23 /pci@3c,0/pci10de,375@b/pci1000,1000@0/sd@7,0 c5t3d0 sd11 /pci@3c,0/pci10de,376@f/pci1000,1000@0/sd@3,0 c5t7d0 sd15 /pci@3c,0/pci10de,376@f/pci1000,1000@0/sd@7,0 c0t1d0 sd25 /pci@0,0/pci10de,377@a/pci1000,1000@0/sd@1,0 c0t2d0 sd26 /pci@0,0/pci10de,377@a/pci1000,1000@0/sd@2,0 c0t4d0 sd28 /pci@0,0/pci10de,377@a/pci1000,1000@0/sd@4,0

hd -R (SMART data)
# hd -R 0 c0t0 1 c0t1 2 c0t2 3 c0t3 4 c0t4 5 c0t5 6 c0t6 7 c0t7 8 c1t0 9 c1t1 10 c1t2 11 c1t3 12 c1t4 13 c1t5 [snip] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 98 98 96 96 95 96 95 95 95 95 95 91 91 91 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23940212 64136331 63714271 77769906 63950789 60284644 63323738 86063623 50456181 64057762 64248194 62234017 64272043 63328043 3700 3697 3697 3700 3699 3697 3699 3698 3791 3789 3791 3792 3793 3791 0 0 0 0 0 0 0 0 0 0 0 0 0 0 98 98 96 96 95 96 95 96 95 95 95 92 91 91 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 437518362 487915548 521469982 555089953 454361114 487915548 504692766 538247199 437518361 487915549 487850012 521469982 437518362 471072795 26 28 30 33 26 28 30 31 25 29 28 30 26 27 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 21 21 22 21 21 21 21 20 21 20 21 20 20 162397417 118980594 119473932 147937438 113649796 119869702 113137454 165014105 187526371 125004952 131054766 158572316 112546961 120678423 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2204 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

In Manufacturing, the threshold "Raw data" is greater than "14" for the followings IDs: 5 Reallocated sector count 196 Reallocation event count 197 Current pending sector count 198 Scan uncorrected sector count And greater than "1" for 199 Ultra DMA CRC error count According to the Drive Vendor, ID #1 (Read Error Rate) is the sampling rate which might fluctuate and it is vendor specific, and NOT TO BE USED screen the disk.

hd -r (SMART data)
3 c0t3 ====== Revision: 10 Offline status 130 Selftest status 0 Seconds to collect 430 Time in minutes to run short selftest 1 Time in minutes to run extended selftest 92 Offline capability 91 SMART capability 3 Error logging capability 1 Checksum 0xaf Identification Status Current Worst 1 Raw read error rate 0xf 100 253 3 Spin up time 0x3 95 95 4 Start/Stop count 0x32 100 100 5 Reallocated sector count 0x33 100 100 7 Seek error rate 0xf 78 60 9 Power on hours count 0x32 96 96 10 Spin retry count 0x13 100 100 12 Device power cycle count 0x32 100 100 187 Uncorrectable Errors for Host 0x32 100 100 189 High Fly Writes 0x3a 100 100 190 Airflow Temperature (WDC) 0x22 68 59 194 Temperature 0x22 32 41 195 Hardware ECC Recovered 0x1a 64 60 197 Current pending sector count 0x12 100 100 198 Scan uncorrected sector count 0x10 100 100 199 Ultra DMA CRC error count 0x3e 200 200 200 Write/Multi-Zone Error Rate 0x0 100 253 202 Data Address Mark errors 0x32 100 253

32/

Raw data 0 0 96 0 77769927 3700 0 96 0 0 555089952 0/ 22 (degrees C cur/min/max) 147942563 0 0 2204 0 0

<Insert Picture Here>

Disk Replacement

Replace a disk

# zpool pool: state: scrub: config:

status zpool1 ONLINE resilver completed with 0 errors on Tue Nov NAME zpool1 raidz1 c5t1d0 c5t5d0 c4t1d0 c4t5d0 c7t1d0 STATE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE faulted ONLINE READ WRITE CKSUM 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

4 18:55:02 2008

errors: No known data errors

Replace a disk
# cfgadm | grep sata | grep disk sata0/0::dsk/c0t0d0 sata0/1::dsk/c0t1d0 sata0/4::dsk/c0t4d0 sata0/5::dsk/c0t5d0 sata1/0::dsk/c1t0d0 sata1/1::dsk/c1t1d0 sata1/4::dsk/c1t4d0 sata1/5::dsk/c1t5d0 sata2/0::dsk/c4t0d0 sata2/1::dsk/c4t1d0 sata2/4::dsk/c4t4d0 sata2/5::dsk/c4t5d0 sata3/0::dsk/c5t0d0 sata3/1::dsk/c5t1d0 sata3/4::dsk/c5t4d0 sata3/5::dsk/c5t5d0 sata4/0::dsk/c6t0d0 sata4/1::dsk/c6t1d0 sata4/4::dsk/c6t4d0 sata4/5::dsk/c6t5d0 sata5/0::dsk/c7t0d0 sata5/1::dsk/c7t1d0 sata5/4::dsk/c7t4d0 sata5/5::dsk/c7t5d0 disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok

Replace a disk

# zpool pool: state: scrub: config:

status zpool1 ONLINE resilver completed with 0 errors on Tue Nov NAME zpool1 raidz1 c5t1d0 c5t5d0 c4t1d0 c4t5d0 c7t1d0 STATE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE READ WRITE CKSUM 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

4 18:55:02 2008

# cfgadm -c unconfigure sata5/1 Unconfigure the device at: /devices/pci@2,0/pci1022,7458@8/pci11ab,11ab@1:1 This operation will suspend activity on the SATA device Continue (yes/no)? yes cfgadm: Hardware specific failure: Failed to unconfig device at ap_id: /devices/pci@2,0/pci1022,7458@8/pci11ab,11ab@1:1

Replace a disk
# zpool offline zpool1 c7t1d0 Bringing device c7t1d0 offline # zpool pool: state: status: status zpool1 DEGRADED One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: resilver completed with 0 errors on Mon Nov 17 00:23:21 2008 config: NAME zpool1 raidz1 c5t1d0 c5t5d0 c4t1d0 c4t5d0 c7t1d0 STATE DEGRADED DEGRADED ONLINE ONLINE ONLINE ONLINE OFFLINE READ WRITE CKSUM 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

errors: No known data errors

# cfgadm -c unconfigure sata5/1 Unconfigure the device at: /devices/pci@2,0/pci1022,7458@8/pci11ab,11ab@1:1 This operation will suspend activity on the SATA device Continue (yes/no)? yes BLUE ready to remove LED is now ON # cfgadm | grep sata | grep disk sata0/0::dsk/c0t0d0 disk sata0/1::dsk/c0t1d0 disk sata0/4::dsk/c0t4d0 disk sata0/5::dsk/c0t5d0 disk sata1/0::dsk/c1t0d0 disk sata1/1::dsk/c1t1d0 disk sata1/4::dsk/c1t4d0 disk sata1/5::dsk/c1t5d0 disk sata2/0::dsk/c4t0d0 disk sata2/1::dsk/c4t1d0 disk sata2/4::dsk/c4t4d0 disk sata2/5::dsk/c4t5d0 disk sata3/0::dsk/c5t0d0 disk sata3/1::dsk/c5t1d0 disk sata3/4::dsk/c5t4d0 disk sata3/5::dsk/c5t5d0 disk sata4/0::dsk/c6t0d0 disk sata4/1::dsk/c6t1d0 disk sata4/4::dsk/c6t4d0 disk sata4/5::dsk/c6t5d0 disk sata5/0::dsk/c7t0d0 disk sata5/1 disk sata5/4::dsk/c7t4d0 disk sata5/5::dsk/c7t5d0 disk connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured unconfigured configured configured ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok

Replace a disk

HDD and Fan Tray LED

Busy, Fault and Ready to Remove LEDs are identical to Thumper

Same physical chassis Same meaning and operation

Sun Fire X4540 Disk Indicators

IPMI interface to indicators is identical to Thumper In addition to standard documentation, suggest referring to previous Thumper TOI
http://panacea/twiki/bin/view/Products/ProdTrainingSunF ireX4500

Replace a disk
# cfgadm | grep sata | grep disk sata0/0::dsk/c0t0d0 sata0/1::dsk/c0t1d0 sata0/4::dsk/c0t4d0 sata0/5::dsk/c0t5d0 sata1/0::dsk/c1t0d0 sata1/1::dsk/c1t1d0 sata1/4::dsk/c1t4d0 sata1/5::dsk/c1t5d0 sata2/0::dsk/c4t0d0 sata2/1::dsk/c4t1d0 sata2/4::dsk/c4t4d0 sata2/5::dsk/c4t5d0 sata3/0::dsk/c5t0d0 sata3/1::dsk/c5t1d0 sata3/4::dsk/c5t4d0 sata3/5::dsk/c5t5d0 sata4/0::dsk/c6t0d0 sata4/1::dsk/c6t1d0 sata4/4::dsk/c6t4d0 sata4/5::dsk/c6t5d0 sata5/0::dsk/c7t0d0 sata5/1::dsk/c7t1d0 sata5/4::dsk/c7t4d0 sata5/5::dsk/c7t5d0 disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured un configured configured configured configured ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok

Replace a disk
# cfgadm -c configure sata5/1 # cfgadm | grep disk | grep sata sata0/0::dsk/c0t0d0 sata0/1::dsk/c0t1d0 sata0/4::dsk/c0t4d0 sata0/5::dsk/c0t5d0 sata1/0::dsk/c1t0d0 sata1/1::dsk/c1t1d0 sata1/4::dsk/c1t4d0 sata1/5::dsk/c1t5d0 sata2/0::dsk/c4t0d0 sata2/1::dsk/c4t1d0 sata2/4::dsk/c4t4d0 sata2/5::dsk/c4t5d0 sata3/0::dsk/c5t0d0 sata3/1::dsk/c5t1d0 sata3/4::dsk/c5t4d0 sata3/5::dsk/c5t5d0 sata4/0::dsk/c6t0d0 sata4/1::dsk/c6t1d0 sata4/4::dsk/c6t4d0 sata4/5::dsk/c6t5d0 sata5/0::dsk/c7t0d0 sata5/1::dsk/c7t1d0 sata5/4::dsk/c7t4d0 sata5/5::dsk/c7t5d0 disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk

This step necessary on x4500 only. x4540 auto-configures on insertion in 5-10 seconds.
connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected connected configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured configured ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok

# zpool pool: state: status:

Replace a disk status

zpool1 DEGRADED One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: resilver completed with 0 errors on Tue Nov 4 18:55:02 2008 config: NAME zpool1 raidz1 c5t1d0 c5t5d0 c4t1d0 c4t5d0 c7t1d0 STATE DEGRADED DEGRADED ONLINE ONLINE ONLINE ONLINE OFFLINE READ WRITE CKSUM 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

errors: No known data errors # zpool online zpool1 c7t1d0 Bringing device c7t1d0 online # zpool status pool: zpool1 state: ONLINE scrub: resilver completed with 0 errors on Mon Nov 17 00:23:21 2008 config: NAME zpool1 raidz1 c5t1d0 c5t5d0 c4t1d0 c4t5d0 c7t1d0 STATE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE READ WRITE CKSUM 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Power Supplies

Two redundant, hotswappable power supplies Unused power supply bay must have a blank filler installed Thumper 220v only Thor 110V option with 3rd PSU expected Dec 2008. Needs new chassis, new PSUs, new PDB.

Thor Back Plate

2 x Redundant Power Supplies (3x for 110v coming soon) 3 x PCI-e slots for IO cards 1 x Serial port 1 x 10/100BaseT Ethernet for SP 4 x 10/100/1000BaseT network port 2 x USB 2.0 ports 1 x VGA port 1 x Compact Flash slot

Thumper Back Plate

1 x PCI slot for Graphic card +SP board 2 x PCI-X slots for IO cards 1 x Serial port 2 x USB 2.0 ports 4 x 10/100/1000 BASE-T network port

Thor System Controller Removal and Installation Removal

Pull extractor level down until it's at a 45 degree angle. Module will slide back Remove with caution system controller is heavy.

Installation
Insert system controller into location and push forward until lever engages. Push lever up to completely engage the connectors. Make sure that module is installed straight and even or system may not power on.

System Controller - Removing

Not seated or bad CPU if any missing

Thumper & Thor Serial Port RS-232 serial interface RJ45 connector Console only, no modem support (no RI, PPP) Connected to ILOM by default Default parameters
9600 baud 8 data bits No parity 1 stop bit No flow control

<Insert Picture Here>

PCI details

Thor PCIe Card Sizing

PCI-e
8-lane (x8) MD2 51.5 mm (2.5 inches) maximum height 169.3 mm (6.7 inches) maximum length

Thumper PCI Card Sizing

PCI, PCI-X
MD2 51.5 mm (2.5 inches) maximum height 169.3 mm (6.7 inches) maximum length

Thor PCIe Slot Locations PCI-e slot 0, 8-lane PCI-e slot 1, 8-lane PCI-e slot 2, 8-lane

Thumper PCI Slot Locations PCI-X slot 0, 64bit, 133Mhz PCI-X slot 1, 64bit, 133Mhz PCI slot, 32 bit 33 Mhz

<Insert Picture Here>

Cooling

Thumper & Thor Cooling & Air Flow Cooler components = greater energy efficiency and reliability

10 hot swappable fans cool HDD CPU's, Memory, PCI, and motherboard

Thumper & Thor Cooling Fans Hot swappable Variable speed 7500 R.P.M. max Top loading Fault/OK LEDs 1.8A / 18W

SP software will control the fan speed and detect a fan failure System can not sustain operation if a fan tray is removed

operate the system with a fan removed for more than 60 seconds

Thumper & Thor Environmentals

Environmental Specifications Operating Temperature 41 95F (5 - 35C) Storage Temperature -4 140F (-20 60C) Humidity 20 90% non-condensing Operating Altitude 0 10,000 feet (0 3048 m) max

<Insert Picture Here>

Service Processor

Thumper Graphic/SP Board Block Diagram

ILOM Service Processor Sun Fire box-level management ConsistentX4000 Family for the entire product line Field upgradeable firmware Connects to all major system components through onboard interfaces Separate mgmt network System mgmt, fan speed, diagnostic LED's

Thumper ILOM

Daughtercard FRU CLI and GUI Industry standard protocols

HTTP(S), IPMI, SNMP, LDAP, SSH, DMTF SMASH

Scale out management using N1SM

ILOM Monitoring and Control Control and Monitoring Fan control Thermal shutdown (CPU or ambient over temperature) Recording events reported by the BIOS (e.g. POST progress, ECC errors) Reset, ACPI-based graceful shutdown, and power control. Control and Monitoring Interfaces IPMI inband (KCS) and out of band (management ethernet) (e.g. using ipmitool) CLI WebGUI SNMP V1, 2c, 3

ILOM Management Interfaces CLI and WebGui: Event logs Configuration of network/user/SNMP/alert/clock Host reset, ACPI-based graceful shutdown, power control SP reset CLI only: Serial console forwarding (via ssh or RJ45 serial port). SP/BIOS upgrade via tftp WebGUI only: Sensors/Indicators SP/BIOS upgrade via Web Download Start javaRemoteConsole

Thor Normal Operation ILOM boot First ILOM boots Service processor boots upon connecting AC Power/insertion/chassis power on. SP runs on auxiliary power Booting takes about 2 minutes Linux boot output appears on serial console. Network: DHCP or static IP, 10/100Mbit (DHCP default) If BIOS is set to "power up upon power failure" the host power will be on during SP boot but the SP holds the host in RESET until the SP is up. ILOM shells (via ssh or RJ45 serial port): root/changeme -> CLI sunservice/changeme -> Linux shell. There are debugging tidbits in /coredump & /var/log

Thor Cooling fan speed

The fan speed is controlled by the Service Processor using Pulse Width Modulation through ADM102x. All fans are controlled with the same PWM frequency so they all run at [approximately] the same speed. The Service Processor uses readings from several sensors to determine the required fan speed: CPU (pN.t_core) Ambient temperature (dbp..t_amb) (See next slide for details) The Service Processor reads fan tachometer outputs and turns on the fan fault LED/Chassis fault led if the RPM is too low

Thor Cooling fan speed con't

The CPU core temperatures are from the on-package thermal-diode reading from CPUs The ambient temperature reading is from the LM75 temperature sensor on top of the disk backplane, behind the 5 chassis fans.

System Details Jumpers

FORCE RECOVERY CMOS clear test points are provided. WRITE ENABLE enables reprogramming of SP mac address by manufacturing. SERIAL OUTPUT enables host versus SP output on serial port (also see /SP/serial/portsharing).
J2802 Clear CMOS & BIOS passwords ipmitool -U root -P changeme -H <spip> chassis bootdev disk clear-cmos=yes No bootblock flash recovery supported in BIOS No clear BIOS password jumper supported in hardware

Jumpers Miscellaneous
BIOS CMOS clear jumper: need to "load optimal defaults" and "save" BIOS recovery jumper: forces update of BIOS from USBfloppy or USB-CD. Alternatively use the SP to reflash.

Power Supplies

PSUs have IPMI FRUID visible via ipmitool. 110* and 220 volt PSUs supported 220v using 2 PSUs at RR 110v using 3 PSU released after RR (THOR ONLY) ILOM considers 1 PSU, or 2 110v PSUs, a degraded/fault condition Ipmitool -U root -P changeme -H <spipaddress> sel elist IPMI SEL log tracks insertion/removal of must CRU/FRU in system Note: a version of ipmitool is available on SP sunservice account for internal engineering.

Normal Operation ILOM Oddities Power failure during SP firmware upgrade may result in an inoperative service processor. socflash.exe, Ipmiflash -I pci, or SIA cdrom, required to recover. Power failure during BIOS upgrade is no problem, just retry. ipmitool retrieves more error logged information than displayed by the WebGUI and CLI. For DRAM and correctable ECC errors, ipmitool tells you which individual DIMM has the problem. Please get ipmitool data, not a web GUI copy and paste nor show commands from the SP!

ILOM RemoteConsole
Remote Console USB keyboard and mouse composite device USB floppy and CD/DVD Video NEW: resizable video window USB devices USB devices appear to be always plugged in. If not redirected, USB media is absent. BIOS should report an "extra" keyboard, mouse, AMI virtual floppy, AMI virtual CD in addition to the USB devices actually plugged into the thor.. Mouse emulation supports two protocols: relative and absolute. Absolute works better. Windows and S10 understand absolute, Linux does not.

FRU ROMs: Service & Board Replacement

When field service swaps mainboard/pdb the technician will need to log in to the Service Processor using the 'sunservice' login and run 'servicetool'. $ servicetool --help provides extensive help for servicetool and it is a verbose, prompt-driven tool. Servicetool is able to: write the system specific FRU information to the motherboard SeePROM if something goes awry and the information is corrupted or lost. refresh the FRUs after a board is swapped, e.g. when the motherboard is swapped, servicetool is used to force the system specific data cached in the pdb to be written back to the motherboard SeePROM.

Sunservice --help

[(flash)root@SUNSP00144F26FCD8:~]# servicetool help Usage: servicetool OPTIONS OPTIONS: --board_replaced=BOARD update FRU information for BOARD after board has been replaced --fru_product_part_number write a new product part number to the mainboard FRU --fru_product_serial_number write a new product serial number to the mainboard FRU --fru_chassis_serial_number write a new chassis serial number to the mainboard FRU --fru_asset_tag write a new asset tag to the mainboard FRU BOARD may be one of: mainboard service_processor

ILOM RKVMS
javaRConsole displays Remote Video in App Window

Graphics Redirected over Ethernet

Video up to 1024x768@60Hz

Management console

Local Mouse and Keyboard

JavaRConsole connected to ILOM over management Ethernet

Thor server

CDROM, DVDROM or iso image Floppy disk or floppy image Keyboard, Mouse, CDROM, and Floppy are seen as USB

Remote Keyboard, Mouse and Storage emulated as USB devices by ILOM

ILOM Javarconsole Supported Platforms Clients: Windows Vista/XP SP2, S10 on x86 or SPARC, various Enterprise Linuxes on x86 Issue: mac os not supported

<Insert Picture Here>

Screen Shots

Thor BIOS Boot Screen

Thumper BIOS Boot Screen

Thor LSI BIOS

Thor LSI BIOS

Factory GRUB boot menu

<Insert Picture Here>

Solaris Specifics

Solaris (thor) Support Solaris10 Update 5 CR 6603801 use fmdump or fmadm faulty instead of fmdump -e. Using LSI IT (non RAID) firmware and raidctl is not supported. S10U5 has generic FMA support for Barcelona. FMA can detect ECC error but cannot locate faulty DIMMs. SunVTS 7.0ps1 is bundled in S10U5.

Thor BIOS/SP Interaction

After AC power cycle, BIOS waits up to 3 minutes for SP to boot (time-out in 10 minutes if no responses from SP) Previous platforms, BIOS always reports an extra keyboard, mouse, AMI virtual floppy, AMI virtual CD in addition to the USB devices actually plugged into the systems. Now, SP only presents devices if redirection enabled in ILOM. BIOS should report BMC Responding at the end of BIOS POST Extensive list of POST progress, Information, and error messages passed to SEL by Error Handling

Thor BIOS boot disks

The BIOS will show maximum only four boot disks in POST When the OS booted, it will load the LSI driver and the entire 48 disks (if installed) will be shown The user can invoke the LSI SCSI utility in POST by pressing the CTRL-C and change the default settings

Thor Default Boot Priority

Compact Flash USB 2.0 (back ports) USB 2.0 (front ports) LSI 0 LSI 1 NICs PCI-e slot 0 PCI-e slot 1 PCI-e slot 2

Th or

BIOS Boot image

Thor BIOS boot disks Each of six on-board LSI controllers connected to 8 disks LSI 0 in bus 02h, dev 0, func 0 connects to ID00-ID07 LSI 1 in bus 03h, dev 0, func 0 connects to ID00-ID07 LSI 2 in bus 04h, dev 0, func 0 connects to ID00-ID07 LSI 3 in bus 41h, dev 0, func 0 connects to ID00-ID07 LSI 4 in bus 42h, dev 0, func 0 connects to ID00-ID07 LSI 5 in bus 43h, dev 0, func 0 connects to ID00-ID07

Thor BIOS boot disks BIOS will show the hard disks in F2 Setup as follow: SCSI: #0200 ID00 SCSI: #0300 ID01

.... ....
LSI SCSI controller in bus 02h, dev 0, func 0, hard disk ID00 LSI SCSI controller in bus 03h, dev 0, func 0, hard disk ID08

Thor Option ROMs (OpROMs)

Legacy PC Architecture leaves 128KB for OpROMs Onboard OpROMs (PXE, AST2000, LSI) use 65KB This leaves 63KB for Option Cards If Option ROM Space Exhausted warning during POST, use F2 Setup to disable OpROMs that are not needed to boot or in DOS User can enable or disable option ROMs for option cards, on-board LSI SCSI controllers and on-board NICs in F2 Setup

BIOS Option ROM

Thor Current Known Issues

http://panacea.sun.com/twiki/bin/view/Products/ProdIssuesSunFir eX4540

Sun Installation Assistant ( SIA ) Thor SW 1.0

What is SIA?
SIA Sun Installation Assistant What it is? Host software for X64 system setup & off-line maintenance Guides customers through common tasks Hardware configuration and firmware updates OS installation: Windows, Linux Driver Installation: Windows and Linux Post OS-installation: Install Sun distributed software Why it is needed? Common customer question: What firmware, drivers and applications do I need, and Where do I get them SIA packages firmware, drivers, applications and makes it easy to deploy them in one easy step during OS installation

How to get SIA software Physical CD with the system Downloadable

http://www.sun.com/systemmanagement/sia.jsp Media options CD ISO USB Thumbdrive Image

Thor SIA Version

2.1.2. 0

I have obtained SIA, what next?

SIA runs on the host and system must be booted into SIA SIA supports a wide variety of boot options CD/DVD drive Virtual CD using CD redirection PXE boot USB Thumbdrive Boot SIA and perform one or more supported server maintenance tasks!

Check with sun for software updates (Optional Step)

Supported Tasks on Thor

OS Support and Install Media

BIOS/ILOM FW Upgrade

Storage Controller FW Upgrade

SP Recovery

Applications Delivered on SIA Hdtool Windows and Linux HERD Linux DCMU Linux Service Tags Hardware registration for Windows and Linux Many others

Option Cards Supported on SIA

Gigabit Ethernet 1-Port SX PCI-X X7286A Gigabit Ethernet 2-Port CU PCI-X X7285A 10GbE 850nmFX(GigaSwift) PCI-X X5544A-4 Infiniband 4X(CX4) PCI-X X1333A-4 LSI53C1020 U320 SCSI 1-Port PCI-X SG-XPCI1SCSI-LM320 2 GB Fibre Channel 1/2-Port PCI-X SG-XPCI1FC-EM2 4 GB Fibre Channel 1/2-Port PCI-X SG-XPCI1FC-EM4-Z Entry Level 2GB FC 1-Port PCI-X(Qlogic) SG-XPCI1FC-QLC-Z FC2 1-Port (Amber 2A, Qlogic) SG-XPCI1FC-QL2 ISP2422 4GB FC 1/2-Port PCI-X SG-XPCI2FC-QF4 EVER CHANGING, Check for SW release on SDLC for latest release.

Known Issues Only detected option cards drivers are installed

Useful support info

Thumper issue page http://panacea.central.sun.com/twiki/bin/view/Products/ProdIssuesSunFireX4500 Thor issue page http://panacea.central.sun.com/twiki/bin/view/Products/ProdIssuesSunFireX4540 SW Release Matrix http://panacea.central.sun.com/twiki/bin/view/Products/SWReleaseMatrix SIA Page http://www.sun.com/systemmanagement/sia.jsp Alias list https://cetwo.sfbay.sun.com/x/SpPpAw