12c ASM: Moving the Existing Cluster Mode to Flex Cluster

In Oracle Flex ASM architecture, Oracle ASM instances can run only a small number of cluster nodes. These ASM instances are connected by all the database instances on the cluster to provide storage access for all these database instances. When an ASM instance fails, the database instances reconnect to other ASM instances for the failed ASM instance. This features supports the platform for variety of application which includes RAC with large number of nodes. There is a new type of network, ASM network with Flex ASM which is used for communication between ASM and its clients. ASM network is accessible from all nodes. In oracle Flex Cluster, the nodes are arranged in two ways i.e. Hub Node and Leaf Node.

Hub Nodes: tightly connected with each other like standard configuration and have direct access to the shared storage.

Leaf nodes: Leaf nodes can be added as many as more and can host different applications other than hub nodes. This type of nodes doesn't require direct access to shared storage instead they can request data from Hub nodes.

The Hub nodes can be run or configured as member of cluster without any leaf nodes but leaf nodes must have Hub node to become member of cluster.

Check current ASM Status
 

[oracle@rac1 bin]$ ./crsctl get cluster  mode status
Cluster is running in "standard" mode

[oracle@rac1 bin]$ ./srvctl status asm -detail

ASM is running on rac1,rac2
ASM is enabled.

[oracle@rac1 bin]$ ./srvctl config asm

ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER

[oracle@rac1 bin]$ asmcmd showclustermode

ASM cluster : Flex mode disabled

oracle@rac1 bin]$ asmca

 

Select ASM Instance tab

click Convert to Oracle Flex ASM

click OK

 

Run the script on local node as a root user :

/u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh

Output:

[root@rac1 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh

PRCR-1086 : resource ora.proxy_advm is already registered

CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.FRA.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'

:

CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

:
:
:

CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac2'
CRS-2677: Stop of 'ora.cvu' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac1'
CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded
CRS-2677: Stop of 'ora.rac2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac1'
CRS-2676: Start of 'ora.rac2.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac1'
CRS-2676: Start of 'ora.scan2.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac1'
CRS-2676: Start of 'ora.scan3.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
:

:

CRS-2676: Start of 'ora.storage' on 'rac2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
Cluster bounced in node rac2
Check the mode again
---------------------------

Check the clustermode again:

--------------------------------------
[oracle@rac1 bin]$ asmcmd showclustermode
ASM cluster : Flex mode enabled

Check Configuration

---------------------------
[oracle@rac1 bin]$ ./srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM

On First Node:

[oracle@rac1 bin]$ ps -ef|grep pmon
oracle   15633     1  0 07:41 ?        00:00:00 asm_pmon_+ASM1
oracle   15985     1  0 07:42 ?        00:00:00 apx_pmon_+APX1
oracle   21959  7848  0 08:00 pts/1    00:00:00 grep pmon

Check the instance status:
-----------------------------------------

[oracle@rac1 bin]$ ./srvctl status database -d orcl

Instance orcl_1 is running on node rac2

So instance is running on only second node based on server policy.

Check on Second Node:

---------------------------------------

[oracle@rac2 ~]$ ps -ef | grep pmon

oracle   14453     1  0 07:43 ?        00:00:00 asm_pmon_+ASM2
oracle   17339     1  0 07:44 ?        00:00:00 apx_pmon_+APX2
oracle   20393     1  0 07:56 ?        00:00:00 ora_pmon_orcl_1
oracle   22522 22466  0 08:17 pts/1    00:00:00 grep pmon

Note: New +APX proxy instances are running on nodes for Flex ASM

Check Network status

-----------------------------------

[oracle@rac2 ~]$ netstat -ntl | grep 1521

tcp        0      0 192.168.2.24:1521           0.0.0.0:*                   LISTEN     
tcp        0      0 192.168.2.12:1521           0.0.0.0:*                   LISTEN     
tcp        0      0 192.168.2.102:1521          0.0.0.0:*                   LISTEN     
tcp        0      0 192.168.0.102:1521          0.0.0.0:*                   LISTEN     

Now for testing I am going to kill ASM on cluster node 2

[oracle@rac2 ~]$ ps -ef | grep pmon

oracle   14453     1  0 07:43 ?        00:00:00 asm_pmon_+ASM2
oracle   17339     1  0 07:44 ?        00:00:00 apx_pmon_+APX2
oracle   19985 22466  0 08:30 pts/1    00:00:00 grep pmon
oracle   20393     1  0 07:56 ?        00:00:00 ora_pmon_orcl_1

[oracle@rac2 ~]$ kill -9 14453 17339

[oracle@rac2 ~]$ ps -ef | grep pmon

oracle   20393     1  0 07:56 ?        00:00:00 ora_pmon_orcl_1

oracle   32397 22466  0 08:31 pts/1    00:00:00 grep pmon
 
Note: RAC database instance on node 2 is still running.

Check the ASM status

-----------------------------
[oracle@rac2 ~]$ crsctl stat res ora.asm -v

NAME=ora.asm

TYPE=ora.asm.type
LAST_SERVER=rac1
STATE=ONLINE on rac1
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=223
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.asm 1 1
INCARNATION=1
LAST_RESTART=11/16/2013 07:38:40
LAST_STATE_CHANGE=11/16/2013 07:42:54
STATE_DETAILS=
INTERNAL_STATE=STABLE
INSTANCE_COUNT=2

LAST_SERVER=rac2

STATE=OFFLINE
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=223
RESTART_COUNT=1
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.asm 2 1
INCARNATION=1
LAST_RESTART=NEVER
LAST_STATE_CHANGE=11/16/2013 08:31:03
STATE_DETAILS=
INTERNAL_STATE=STARTING
INSTANCE_COUNT=2
 

Check again

--------------------
[oracle@rac2 ~]$ ps -ef | grep pmon
oracle     927     1  0 08:31 ?        00:00:00 asm_pmon_+ASM2
oracle    1842     1  0 08:31 ?        00:00:00 apx_pmon_+APX2
oracle    2783 22466  0 08:35 pts/1    00:00:00 grep pmon
oracle   20393     1  0 07:56 ?        00:00:00 ora_pmon_orcl_1

Note: ASM has restarted and the database instance has remained stable.

Flex Cluster Role Management

------------------------------------------
[root@rac1 bin]# ./crsctl status server rac1 -f
NAME=rac1
MEMORY_SIZE=3854
CPU_COUNT=1
CPU_CLOCK_RATE=2
CPU_HYPERTHREADING=0
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=1
SERVER_LABEL=
PHYSICAL_HOSTNAME=
STATE=ONLINE
ACTIVE_POOLS=Generic ora.TESTP
STATE_DETAILS=
ACTIVE_CSS_ROLE=hub

 

Change the 1st node as a leaf node:

[root@rac1 bin]# crsctl set node role leaf

CRS-4408: Node 'rac1' configured role successfully changed; restart Oracle High Availability Services for new role to take effect.

[root@rac1 bin]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'

:

:

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac1 bin]# crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

Check again
----------------------

./crsctl status server rac1 -f

NAME=rac1

MEMORY_SIZE=3854
CPU_COUNT=1
CPU_CLOCK_RATE=2
CPU_HYPERTHREADING=0
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=leaf
RESOURCE_USE_ENABLED=1
SERVER_LABEL=
PHYSICAL_HOSTNAME=
STATE=ONLINE
ACTIVE_POOLS=Generic ora.TESTP
STATE_DETAILS=
ACTIVE_CSS_ROLE=leaf