Cluster network name resource 'Cluster Name' failed registration of one or more associated DNS name event ID 1196 and Event ID 1119

Problem:

After setting up windows 2008 Cluster with SQL and Exchange 2007, the following event logs are showing on the event log of the both clusters. The errors are not causing any fail over or operational issues but interesting to investigate and needs resolution

Inside technical info:

Below errors indicating that, the Active cluster node is not able to register the "Cluster name resource" into its own configured DNS server. Basically this is what is happening. The Cluster does have resource called Cluster network name resource. This is one of the resources must be created prior to setting up a cluster and it is being created manually in the DNS database. The cluster active node will own this resource and the active node production network interface will go to its configured DNS and will try to register the cluster name resource record to the DNS database itself. In this example the DNS is refusing the registration, because DNS knows the server does not own the resource name, and hence it is not allowing the active role node to register it.

What does cluster network name resource do?

"The name of the SQL Server is defined by the network name cluster resource, and that name will be used by applications and end users to connect to the failover instance; together these resources represent a logical Windows Server on the network, while running across one or more real Windows Server computers. A failover instance of SQL Server is an item that is created during SQL Server setup; it is not provided by Windows Server."

Root Cause:

When static record got created the option "allow any authenticated user to update DNS records with the same owner" was not selected. Therefore the Cluster nodes (active node) who will own the cluster name resource won't be able to register this resource record to the DNS database behalf of the resource records itself.

Solution:

Go to DNS, find the record ( A & Pointer record) for the cluster name resource.

• Make a right click
  
• Go to properties
  
• In the security make sure the "Authenticated users" are included
  
• Make sure it has "Write: rights and Special permissions
  
• Click Advance, locate authenticated users, and click edit
  
• Make sure, Write all properties, Read permissions, All Validated Writes selected
  
• Click okay tree times to exit
  

After investigating further, you will notice there are no more errors on the System logs in this regard

Log Name: System Source: Microsoft-Windows-FailoverClustering Date: 10/23/2008 9:30:35 PM Event ID: 1196 Task Category: Network Name Resource Level: Error Keywords: User: SYSTEM Computer: MCCNPWINSQL02.smtp25.org Description: Cluster network name resource 'Cluster Name' failed registration of one or more associated DNS name(s) for the following reason: DNS operation refused. Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-FailoverClustering" Guid="{baf908ea-3421-4ca9-9b84-6689b8c6f85f}" /> <EventID>1196</EventID> <Opcode>0</Opcode> <Keywords>0x8000000000000000</Keywords> <TimeCreated SystemTime="2008-10-24T01:30:35.648Z" /> <EventRecordID>74214</EventRecordID> <Correlation /> <Execution ProcessID="1956" ThreadID="20004" /> <Channel>System</Channel> <Computer>MCCNPWINSQL02.smtp25.org</Computer> <Security UserID="S-1-5-18" /> <Data Name="ResourceName">Cluster Name</Data> <Data Name="StatusString">DNS operation refused. </EventData> </Event>

Oz ozugurlu

MVP (Exchange)

MCITP (EMA) , MCITP (EA ) MCITP(SA),

MCSE (M+,S+) MCDST, Security+, Server +,Project+

Blog: http://www.smtp25.blogspot.com

Windows 2003 to windows 2008 some of the Cluster improvements no more upgrade with CLUSTERS in windows 2008

I heard (Cluster setup steps, never counted there were 23 steps) to setup cluster in windows 2003. The new cluster has couple steps to get the cluster up and running.The windows 2008 brought new Cluster MMC 3.0 so ClueAdmin is gone. Windows 2008, the clustering is being called "Failed over Clustering"

Cluster logging (Text Logging is gone along with ClueAdmin), the new MMC is the tool to use for Windows 2008 Clusters, there is a tool called Validate tool build into the Cluster MMC. This toll provides multiple tests.

What to do when cluster is hang, enable User Mode Hang Detection

• The following symptoms typically indicate that the cluster node has stopped responding: • You can confirm IP connectivity to the server that is hanging by pinging it.
  
• You cannot successfully establish a connection to the server by using the net use command.
  
• You cannot successfully connect to the server by using a Terminal Services client.
  
• You can move the mouse pointer when you log on locally to the server.
  
• You cannot start programs or utilities when you are logged on locally to the server
  

KB is here

Of course this is build right into windows 2008 architect. There is also reporting function build into new cluster configuration management.

Cluster signature managements over better GUI in windows 2008. GPT is supported. New Symantec Foundation for Windows will be supported. The DHCP is supported now. Any network cluster resource will be able to use DHCP support, I am not sure if anyone would really use this because this is supported, I personally would not (-:, but this is supported in windows 2008 cluster configuration just to let you know. NetBIOS is not needed anymore in windows 2008. The Cluster GUI is build into MMC 3.0, so windows 2008 is promising no MORE hourglass simply because the ClueAdmin is gone, and MMC cannot crash (-:

The cluster migration tool will make thing easy, this tool will assist for cluster configurations from one cluster to another one.

Geographically dispersed clusters

Allow cluster nodes to communicate across network router (Wow), configurable heartbeat timeout, no more cluster distance limitations (Wow).

The Cluster management is finally getting retired to be the rocked scientist job to regular admin type of job. The windows 2008 CSA runs the service account as local System which is very nice. No more worry about cluster special account.

I am very excited to expose myself into new cluster management MMC and tolls. Hopefully windows 2008 servers will get deployed fast into most of the network and we will have opportunity to explore the new futures in production environment

Best,

Oz Ozugurlu
Systems Engineer
MCITP (EMA), MCITP (SA),
MCSE 2003 M+ S+ MCDST
Security+Project+ Server+

oz@SMTp25.org
http://smtp25.blogspot.com (Blog)

Microsoft cluster service components

We are in the middle of standing up new clustered exchange servers into our new data center. Part of the process is to build the new cluster servers and move mailboxes over to new exchange server. We probably will build a new cluster within a year and go for exchange 2007. Knowing the cluster components are critical therefore I have decided to post below table here at my blog. Microsoft Cluster Service is consisting of several device drivers and services. These modules reside on top of the Operating system (Windows enterprise edition) and behave as a service. The cluster service provides basic functions that the operating system needs in order to support clustering

• Understanding each component is crucial going trough setting up and managing clusters.
  
• It makes daily admin life easier in my opinion.
  

Component	Role/Function
Node Manager	Maintains resource group ownership of cluster nodes based on resource group node preferences and the Availability of cluster nodes.
Resource Monitor	Utilizes the cluster resource API and RPCs to maintain communication with the resource DLLs. Each monitor runs as a separate process.
Failover Manager	Works in conjunction with the resource monitors to manage resource functions within the cluster such as failovers and restarts.
Checkpoint Manager	Maintains and updates application states and registry keys on the cluster quorum resource.
Configuration Database Manager	Maintains and ensures coherency of the cluster database on each cluster node that includes important cluster information such as node membership, resources, resource groups, and resource types.
Event Processor	Processes events relating to state changes and requests from cluster resources and applications.
Membership Manager	Manages cluster node membership and polls cluster nodes to determine state.
Event Log Replication Manager	Replicate system event log entries across all cluster nodes.
Global Update Manager	Provides updates to the Configuration Database Manager to ensure cluster configuration integrity and consistency.
Object Manager	Provides management of all cluster service objects and the interface for cluster administration.
Log Manager	Works with the Checkpoint Manager to ensure that the recovery log on the cluster quorum disk is current and consistent.

Best

Oz ozugurlu

You cannot remove Cluster Service

You cannot remove the Cluster service, but you can return it to an unconfigured state. You have evicted a node from your Cluster and you want to create a new cluster add the node as a member of the new cluster; However when you try to do it you are getting error, indicating the node is already a member of another Cluster.

• Stop Cluster service
  
• We need to perform a little clean up before we add the server into the new cluster as a new node
  
• My server name is Ex1
  

I will go to RUN menu and type below command

Cluster node ex1 /forcecleanup

Here is Microsoft KB: 282227

The files for the Cluster service are installed, by default, on computers that run either, Windows Server 2003, Enterprise Edition or Windows Server 2003, Datacenter Edition. In earlier versions of Windows, this feature had been in the Add/Remove Programs tool.

You cannot remove the Cluster service, but you can return it to an unconfigured state:

• Start Cluster Administrator (CluAdmin.exe).
  
• Right-click the node, and then click Stop the Cluster service.
  
• Note: Do not perform this step if this server is the last node in the cluster.
  
• Right-click the node, and then click Evict Node.
  

This step returns the cluster to its original unconfigured state. You can re-add it later to the same or to a different cluster. If you cannot start the Cluster service, or if you have trouble removing the node, you can manually unconfigure the Cluster service:

Run the Cmd.exe program to open a command prompt.

At the command prompt, type cluster node nodename /forcecleanup

Note: If the Cluster service does not exist in the registry, the command does not respond. To create a place holder, type the following line at the command line, and then press ENTER:

sc create clussvc

This will perform the clean up and you will be able add this server as new node to the new cluster

Best,

Oz Ozugurlu

Cluster Exchange 2003 & EXRES.DLL administrator should not stop a service from the command-line

Should we use windows services to manage Exchange related services? This has been asked few times, which made me to research online resources. I have decided to post TechNet post without making many changes I hope below Microsoft article will make all of us understand the key DLL files (EXRES.DLL and some of the internal functions such as IsAlive function, and ExchangeOpen functions)

 

• Exchange 2003 is cluster-aware and provides its own cluster resource DLL, named EXRES.DLL, for communication and interaction with the Windows Cluster service. The Windows Cluster service communicates through Resource Monitor to EXRES.DLL, and EXRES.DLL then communicates with the Exchange components. EXRES.DLL translates the cluster actions into actions for Exchange-related services. EXRES.DLL also monitors the stopping of these resources and notifies the Cluster service if the operation is unsuccessful. The following figure illustrates the relationship between EXRES.DLL and the Cluster service
  
• In a cluster, the Cluster service is responsible for starting and stopping Exchange services through EXRES.DLL. Because of this, an administrator should not stop a service from the command-line, the Windows Services snap-in, a Resource Kit tool, or a third-party application.
  
• When you stop a service outside of the Cluster, the IsAlive call to that service fails, causing the Cluster service to attempt a restart of the stopped service. The IsAlive function returns the last value that was pooled from the resource health-monitoring thread.
  
• The LooksAlive function has the same implementation as IsAlive. The Looks Alive function is not called, because the EXRES.DLL provides resource-failure event handles to the cluster Resource Monitor that indicates when a resource fails.
  
• The health-monitoring thread checks resources every ten seconds. This resource check cannot be configured. EXCLUADM.DLL provides the interface associated with Exchange and cluster-specific wizards
  

 

 ExchangeOpen/ExchangeClose Functions

The ExchangeOpen and ExchangeClose functions are called whenever the Exchange resources are moved to the current node. Inside the ExchangeOpen functions, memory is initialized or allocated for the basic information of the resource. These functions also handle the loading and unloading of DLL files that are used by the Resource DLL. This process is controlled by reference counters.

ExchangeOnline and ExchangeOffline Functions

The ExchangeOnline and ExchangeOffline functions create new OnlineWrapperThread and OfflineWrapperThread worker threads and immediately return an ERROR_IO_PENDING to the cluster Resource Monitor. When the wrapper thread returns a failure to Resource Monitor, Resource Monitor tries to restart the resource two additional times. During these restart attempts, the ExchangeOnline function determines if the previous online/offline thread is returned. If the online/offline thread is not returned, the ExchangeOnline function restarts.

For the OfflineWrapperThread, if the ExchangeOffline thread does not return in the PendingTimeOut limit for the Store, System Attendant, and MTA resources, Resource Monitor ends the corresponding process.

To prevent situations in which RPCs hang, these two worker threads create the real online/offline thread. The wrapper threads act as monitors for the online/offline thread and use timers to monitor the operation of the online/offline thread. If the online/offline thread does not return in the PendingTimeOut value that was set in Cluster Administrator, the wrapper thread determines that the operation is unsuccessful, and sets the resource's state as failed. It then returns an error. The only exceptions to this behavior are for upgrade operations or for store resource online operations. In these two cases, the wrapper thread waits for an upgrade to complete or for a store resource to go online without timing out.

 

The ExchangeOnlineThread and ExchangeOfflineThread service the following Exchange resources:

• Micrsoft Exchange System Attendant service, Microsoft Exchange Information Store service, and Routing   Each of these resources starts the service (if it is not already started), and then makes RPC calls to each of the services, instructing them to start or stop the corresponding Exchange Virtual Server.
  
• Protocol Resources   Protocol resources set the command bit in the IIS metabase, and then start the service, if it has not already started. The corresponding service picks up the command and starts or stops the virtual server. The only exception to this is the SMTP virtual server. In this case, the whole SMTP service must be stopped when a virtual server instance is taken offline. The IsAlive function checks for other virtual servers running on the same physical computer, detects that the underlying SMTP service is stopped, and then restarts the virtual server.
  

ExchangeIsAlive and ExchangeLooksAlive

The ExchangeOnline function returns an event handle to Resource Monitor, and Resource Monitor then stops calling the ExchangeLooksAlive function. Instead, the Resource Monitor calls the ExchangeIsAlive function at intervals set in the Cluster Administrator Looks Alive polling interval. Whenever a resource is online, EXRES.DLL creates two threads to monitor the status of that resource. The first thread, named ExchangeIsAliveMonitor, checks the status of the resource every ten seconds by waking the second thread, named ExchangeCheckIsAliveWrapper. ExchangeCheckIsAliveWrapper performs the actual IsAlive checking. If the ExchangeCheckIsAliveWrapper thread does not return in the PendingTimeOut limit, or if the IsAlive check is unsuccessful, the ExchangeIsAliveMonitor thread signals a failure event for the particular resource to Resource Monitor. The ExchangeIsAlive function returns the status of the last IsAlive check.

ExchangeTerminate

This function ends the existing ExchangeIsAliveMonitor thread. For the Exchange store, System Attendant, and MTA resources, it also performs the corresponding offline procedure to make sure that the database is dismounted. If the offline procedure does not complete successfully in the PendingTimeOut limit, EXRES.DLL also terminates the corresponding process.

Creating Resources

The user creates a resource first, and then creates a call to EXCLUADM.DLL, prompting for information. EXCLUADM.DLL obtains the required information for the resource and passes it to the Cluster service. Resource Monitor creates a call to the ExchangeResourceControl function in EXRES.DLL. This call contains the information that was passed from EXCLUADM.DLL and Clusctl_Resource_Set_Private_Properties as the control code. ExchangeSetPrivateResProperties is used to handle this control code. It first saves the information to the cluster database under the Parameters registry key for that resource, and then makes a call to EXSETDATA.DLL to have EXSETDATA.DLL create objects in Acitve Directory. In some cases, a problem might occur and some configuration information might be modified in Exchange System Manager, without synchronizing the changes with the cluster database.

http://technet.microsoft.com/en-us/library/bb124768.aspx

Best

Oz Ozugurlu

Lost Cluster service account password here how to reset it without downtime

Yes you heard right. I am trying to bring another node into existing cluster and I am at the point where I need to supply the Cluster service account password, I am not able to figured out what the password was set too. Of course the person who set the cluster Exchange did not save or document the password for job security reasons and he is no longer at this place. I desperately go over the entire existing password save programs for this company and the result no luck. This Cluster environment is pretty large and there is no room for mistake, so being cautious we ended up calling PS support for Microsoft.

cluster /cluster:nhqCluster /changepassword /skipdc

My cluster name is nhqCluster so; replace this with your cluster name. This command line works like a charm and replaces the password with the new password you will be supplying beside this method I have also figured out, following way works as well. Stop Cluster services on all nodes, go to services make sure cluster service is stopped. Go to properties and reset the password on all the nodes, go back and restart the services. This is involving stopping the cluster; the command line won't cause any interruption on the cluster

Steps:

• First we will change/reset the password for cluster service account in the Active directory.
  
• On every cluster node, click Start, click Run, type services.msc, and then click OK.
  
• In the Services (Local) list, double-click Cluster Service.
  
• Click the Log On tab and then type the Cluster service account in the account box by using the Universal Naming Convention (UNC) format. For example, type an account name that is similar to the following: SMTp25\ clusteraccount
  
• Type the correct account password in the Password box and in the Confirm password box, and then click OK.
  
  
  
• When you receive the following message, click OK: The new logon name will not take effect until you stop and restart the service.
  
• Under Cluster Service, click the Restart link.
  
• Then follow the article KB305813
  
• If it doesn't work the we need to follow the article KB269229 How to manually re-create the Cluster service account
  

Microsoft Links

• How to manually re-create the Cluster service account
  
• http://support.microsoft.com/kb/269229/
• How to change the Cluster service account password
  
• http://support.microsoft.com/kb/305813/
• You receive a "System error 1315" error message when you try to change the Cluster Service account password by using the Cluster.exe command in Windows Server 2003
  
• http://support.microsoft.com/kb/873024
  
  
  

Best regards

Oz Ozugurlu