Upgrade of NSX Controllers to 6.3.4 will not retain syslog configuration

The other day at a customer I noticed that after we had upgraded the NSX environment (NSX Manager and NSX Controllers) from 6.2.4 to 6.3.4 the syslog configured on the NSX Controllers was lost and not retained after the upgrade.
One reason for this I suspect is that with version 6.3 and up on NSX the OS have changed to Photon OS. Thus the requirement to redeploy, delete the old NSX Controllers and create new ones based on the new OS.

Vmware says that the only supported method on configuring the syslog server on the NSX controllers is through the NSX API. One would believe that when you upgrade NSX Controllers that NSX Manager during redeployment of the NSX Controllers would push the old configuration of the old NSX Controllers via API over to the new NSX Controllers automatically.

I cannot myself find any related documentation from VMware regarding this. So I wanted to write a post to remind you to POST new syslog configuration for the NSX Controllers once the upgrade is completed.

Here is how it looks like on the NSX Controller after the upgrade:

Before you start the upgrade of the Controllers be sure to save the current syslog configuration from NSX Controllers so that it is easy to add that information back in the newly deployed NSX Controllers.

This is done by issuing a GET command and finding all the NSX Controller ID names against the NSX Manager:
GET https://nsx-mgr/api/2.0/vdn/controller

Then we fetch the syslog information from the NSX Controllers in this example controller-5 with:
GET https://nsx-mgr/api/2.0/vdn/controller/controller-5

We store this info in a place to remember ūüėČ

Next we upgrade our NSX Manager and NSX Controllers as described in the Vmware Docs.

Next Step after the components are upgraded and everything looks to be OK in the GUI, we go ahead and POST back our Syslog configuration to the new NSX Controllers. Make sure to run the GET controllers again to find the new controller-id names for the new deployed machines since they have changed after the redeployment.

We do a POST against NSX Manager with the following and repeat for all NSX Controllers:
POST https://nsx-mgr/api/2.0/vdn/controller/controller-9/syslog
HEADER: Content-Type: application/xml

Now head over to the Syslog receving server, vRealize Loginsight is what I use, and check that the logs are once again beeing recieved from the Controllers.

Happy NSX-ing, Happy Channuka, X-Mas and New Year….

Passed the VMware Certified Implementation Expert 6 ‚Äď Network Virtualization Exam

Sorry for not posting anything in a while, I have been studying alot for the¬†VMware Certified Implementation Expert 6 ‚Äď Network Virtualization Exam,¬†and yesterday I passed it with 383/500 points needed.

I just recieved my 2 new badges from Acclaim regarding the certifications that I now have recieved from Vmware.

Here I will descibe my experience in studying for the exam and what it takes for you to pass it yourself.

I read the awesome blog from vZealand.com and if you haven’t done so go and do it now. I would like to thank Clinton Prentice for his awesome VCAP6-NV study guide that in very good detail goes through the Vmware Blueprint for every chapter that is needed in order to study to the exam.

I also as you have read in earlier posts by me setup my own vCloud Director vAPP with a full Cross vCenter NSX deployment that I have used a lot in practicing and studying for the exam.

Problems with installing NSX Components on Hosts without correct DNS set.

Today I was encountered with a problem that took me a while to understand.

I was going to prepare a Compute Cluster with the NSX Components, vxlan and vxlan-sip vibs in NSX Manager.

The problem was that it always failed and did not want to install the vibs. The errors that were displayed are shown below.

After a closer inspection in the event logs it was saying that the hosts were unable to access the VIB Modules at the address of my NSX Manager. hmmmm, wierd!

So I started digging and tried to install the VIBs manually see Link, but still without luck of getting the job done.

I then SSH onto each of the hosts and did a NSLOOKUP, and found it strange that I did not have a DNS resolution for the NSX Manager on the ESXi-hosts.

So I went into the TCP/IP Configuration  on the hosts and noticed that an incorrect DNS IP was set on the hosts. The DNS server is but it was set to that is the Default Gateway address. So changed that to the correct one as shown below:

Performed a new nslookup on the hosts, that now showed the correct name for my NSX manager:

As soon as this was completed I jumped back to; NSX Installation of Components and pressed Resolve and look and behold the problem was gone and my components installed successfully. Seems that when the hosts downloads the VIBs from the NSX manager they also need to be able to resolve the name of the IP it connects to even though it goes against the IP of the NSX Manager.

That is is for this post Yaay! And don’t forget to have the DNS set correctly on your hosts!

Part II – Setup full vCenter with NSX Environment vAPP in vCloud Director

In this continuing post I will continue with the installation of the second DC site called DC2.

Here is the overall picture of how it will look like when done in vCenter.

Part I – Setup full vCenter with NSX Environment vAPP in vCloud Director

In this Blog post I will setup a full vCenter with NSX Environment.
The Setup is going to contain
1 Domain Controller based on Windows Server 2016
1 RDS Remote Desktop Server used for maintaining the environment
2 Management ESXi hosts
2 Compute ESXi hosts
1 Openfiler iSCSI VM used to provision shared storage to ESXi hosts.

Inside the ESXi hosts I will the setup:
1 VCSA 6.5 – Vcenter Server Appliance 6.5
1 PSC – Platform Service Controller
1 NSX Manager
3 NSX Controllers

The purpose of doing this is to be able to have a fully usuable vAPP Datacenter (DC1) lab environment for training and learning about Vmware NSX.
In the continuation in another blog I will clone the vAPP created and setup a second vAPP Datacenter called DC2. This is for connecting the two DCs togheter to test out the Cross vCenter with NSX design that I will discuss later on.

The brief design of my vAPP will look like this:

Random ESXCLI commands

ESXCLI Commands:

Here are some random esxcli commands that are useful when working with vSphere and Hosts.

The commands also are very good to learn when studying to the VCAP6-DCV Deploy, Vmware Certified Advanced Professional 6 certification.

add-esxsoftwaredepot c:/files/esxi6.0.zip
new-deployrule -Name ‚ÄĚdeployment‚ÄĚ -Item ‚ÄĚESX-image‚ÄĚ, ‚ÄĚStaging host‚ÄĚ -pattern ‚ÄĚipv4=‚ÄĚ

add-deployrule -Deployrule ‚ÄĚdeployment‚ÄĚ

get-deployruleset -active
 set-executionpolicy unrestricted

add-esxsoftwaredepot c:/depot/vib-offline_bundle.zip

add-esxsoftwaredepot c:/depot/esxi6.5-offline_bundle.zip

 new-esximageprofile -cloneprofile original_profile_name -name "profile_name"
 add-esxsoftwarepackage -imageprofile "profile_name" -softwarepackage package_name -Community
 export-esximageprofile -imageprofile "profile_name" -filepath c:/depot/esxi-profile_name.iso -exporttoiso ‚Äďforce

Vmware Update Manager Download Service:
vmware-umds -S ‚ÄĒenable-host ‚ÄĒenable-va
vmware-umds -S ‚ÄĒpatch-store c:/Patches
vmware-umds -S ‚ÄĒadd-url https://host_url/index.xml ‚ÄĒurl-type HOST
vmware-umds -S ‚ÄĒadd-url https://appliance_url/index.xml ‚ÄĒurl-type VA
vmware-umds -D
vmware-umds -E ‚ÄĒexport-store C:/vmware_download

VMFS resignature:
esxcli storage vmfs snapshot list
esxcli storage vmfs snapshot resignature -l ’volume_name’

esxcfg-mpath -l
esxcli storage core claimrule list
esxcli storage core claimrule add -u -t location -A vmhba2 -C 0 -T 0 -L 0 -P MASK_PATH
esxcli storage core claimrule load
esxcli storage core claimrule run
esxcli storage core claiming reclaim -d naa.010101010100101

ATS SCSI locking:
esxcli storage vmfs lockmode list
esxcli storage vmfs set -a -l VMFSlabel -u VMFSUUID
esxcli storage vmfs set -s -l VMFSlabel -u VMFSUUID

Custom TCP/IP stack
esxcli network ip netstack add -N=‚ÄĚCustomTCPstack‚ÄĚ

Configure vSS med CLI:
esxcli network vswitch standard list
esxcli network vswitch standard add -v vswitch0 -p 10
esxcli network vswitch standard portgroup add -p testpg -v vswitch0
esxcli network  vswitch standard uplink add -u vmnic1 -v vswitch0
esxcli network vswitch standard portgroup set -a vmnic1 -p testpg
esxcli network ip interface add -i vmk1 -p testpg
esxcli network ip interface ipv4 set -t static -ipv4= -n -i vmk1
esxcli network vswitch standard set -mtu=9000 -v vswitch0

esxtop -W
esxtop -b -d 2 -n 1 > out.csv
vm-support -p -d 2 -i 1
tar xvf esxi.tgz
esxtop -R esxidirectory
vscsiStats -l
vscsiStats -s -w 90909

Create a XaaS in vRealize Orchestrator for vRealize Automation and NSX

This week I have dedicated myself of how to create a Workflow in vRealize Orchestrator inorder to create a XaaS, Anything as a Service blueprint in vRealize Automation.

The problem is this: I have created a multi-machine blueprint in vRA see previous post that creates Windows VMs behind an Edge Loadbalancer. Now I want to make sure that it is only possible to RDP to the VMs by using only the IP address to the LB VIP. So together with NSX micro segmentation make sure to block all RDP connections directly to the VMs and only accept to the LB VIP.

In vRO I have created a REST-API connection against the NSX Manager. Let’s call it nsxmanager.local

After the NSX Manager has been registered in vRO. It is time to create a workflow with scriptable tasks. The scriptable tasks are written in JavaScript together with REST API to fetch information with GET command and to create items with the POST command to the NSX Manager API.

A great thing to have in hand when doing this kind of work is having the VMware NSX Rest API documentation available here is the link.

Also to learn alot on how to work with vRO, Javascript, XML and REST API read this whitepaper from Vmware that describes the construction of the scripting Automation Leveraging NSX REST API. Here is the link to the website where the document can be found plus additional information regarding the subject.

And I also a side from working in vRO am using a standalone REST API client, called Insomnia. So that I am able to test and fetch information of the objects I want to get or post info about as I’m trying to build my code. Here is the link to that application.

Workflow Creation Time:

Next I have Created a new Workflow in vRO that is called ‚ÄĚCreate LB IPset, Security Group and Security Policy‚ÄĚ.
The Workflow will be built with four scriptable tasks each doing its own separate thing as described in these bulletpoints.

  • Retrieve Edge
  • Add Edge LB IP to IPSet
  • Create Securitygroup with the name of the Edge LB
  • Add the Securitygroup to a Policygroup

Here is what I want to do in each Scriptable Workflow.


Retrieve Edge:

Retrieve the Names and Edge IP of the Edge Loadbalancer from NSX with GET command in REST API.

Add Edge LB Virtual IP to IPSet:

Pass the Edge LB IP name and IP onward and create a new IPSet with the name of the Edge LB and add the IP from the Edge in the IPSet.

Create Securitygroup with the name of the Edge LB:

Pass along the IPset Object ID and IPset name to create a Securitygroup with the name of the IPSet and add the IPset ObjectID to the Security Group Created.
This is done with the POST REST API command.

Add the Securitygroup to a Policygroup:

Last GET the name of the PolicyGroup called ‚ÄĚint30 Security Policy‚ÄĚ and modify the XML information. and add the Securitypolicy for the Edge IPSet to the Policy.
The Rule that will be created will contain the SecurityGroup ID, the Application ID and Name. For RDP. And set a firewall rule to allow ANY against the Security Group.

My First Scripting task “Add Edge IP to IPSet” looks like this:

var request = restHost.createRequest(“GET”, “4.0/edges/”); //Get all Edge‚Ä®request.contentType = “application/xml”;‚Ä®System.log(“Check all Edge – Request URL: ” + request.fullUrl); //Display request URL‚Ä®var response = request.execute();
if(response.statusCode != 200) throw(“Could not GET edge list ! : ” + response.statusCode + “\n” + response.contentAsString);‚Ä®var edgelistXML = new XML(response.contentAsString); //Convert response as String‚Ä®//System.log(edgelistXML)‚Ä®// Get EdgeSummary node‚Ä®var EdgeList = edgelistXML.edgePage.edgeSummary;
//Check for EdgeID provisioned by vRA.
var EdgeID = [];
for each (var edge in EdgeList) {‚Ä®if (edge.description.text() == “Edge provisioned and managed by VMware vRealize Automation”) {‚Ä®EdgeID.push(edge.objectId.text().toString());‚Ä®}‚Ä®}‚Ä®if (EdgeID.length == null) throw(“No Edge found”);‚Ä®System.log(“Edge-Id found :”+EdgeID);
//Check for VIP‚Ä®var EdgeIP = [];‚Ä®var EdgeName = [];‚Ä®for each(var ID in EdgeID) {‚Ä®var request = restHost.createRequest(“GET”, “4.0/edges/”+ID);‚Ä®request.contentType = “application/xml”;‚Ä®System.log(“Check VIP on Request URL: ” + request.fullUrl);‚Ä®var response = request.execute();‚Ä®if(response.statusCode != 200) throw(“Could not GET edge from ID ! : ” + response.statusCode + “\n” + response.contentAsString);‚Ä®var edgeconfXML = new XML(response.contentAsString);‚Ä®EdgeIP.push(edgeconfXML.features.loadBalancer.virtualServer.ipAddress.text().toString());‚Ä®EdgeName.push(edgeconfXML.name.text().toString());‚Ä®}‚Ä®System.log(EdgeIP);

The next Scripting task looks like this:

//This Task takes the Edge Name and Edge IP and Creates the IPSet in NSX
description= EdgeName + ” LoadBalancer enabled Edge IpSet in VRA”;
//Set the name for the IP Set as IP Set appended to the EdgeName
ipSetName= “IP Set-“+ EdgeName;
ipSetValue= EdgeIP;
System.log(EdgeName +” Loadbalancer VIP is: ” +EdgeIP);
// Prepare the HTTP body as an string with the required
stringbody='<ipset><type><typeName>IPSet</typeName></type><description>’ + description + ‘</description><name>’ + ipSetName + ‘</name><value>’ + ipSetValue + ‘</value><inheritanceAllowed>false</inheritanceAllowed></ipset>’;
// Uncomment the next line to dump the generated string to the debug
//System.debug(“Generated string is: ” + stringbody);
// HTTP POST request is prepared using the URL: /api/2.0/services/ipset/scopeId
// Note that “/api” is not shown in the request URL as it’s already set in the endpointconfiguration
// scopeId is globalroot-0
var request = restHost.createRequest(“POST”, “/2.0/services/ipset/globalroot-0”, stringbody);
request.contentType = “application/xml”;

The third looks like this:

//This Scriptable task creates a Securitygroup based on the name of the IPSet name and after the Securitygroup is created adds the
//IPSet to the Security Group.
// GET info of IPSET from https://nsx.rtsvl.local/api/2.0/services/ipset/scope/globalroot-0
var request = restHost.createRequest(“GET”, “2.0/services/ipset/scope/globalroot-0”); //List all the IPSets
request.contentType = “application/xml”;
System.log(“Check all IPSets – Request URL: ” + request.fullUrl); //Display request URL
var response = request.execute();
var IpSetlistXML = new XML(response.contentAsString); //Convert response as String
var IpSetList = IpSetlistXML.ipset;
var IpSetID = [];
for each (var ipset in IpSetList) {
if (ipset.name.text() == ipSetName) {
if (IpSetID.length == null) throw(“No IpSetID found”);
System.log(“IpSet-Id found: “+IpSetID);
//Create a SecurityGroup based on name on IPset
stringbody='<securitygroup><objectId>’ + ipSetName + ‘</objectId><objectTypeName>SecurityGroup</objectTypeName><vsmUuid>4216ECB4-6001-0631-7E0A-8A052CEA2062</vsmUuid><nodeId>5acf1029-4d59-46cd-84d6-b3edbbeec2bf</nodeId><revision>0</revision><type><typeName>SecurityGroup</typeName></type><name>’ + ipSetName + ‘</name><scope><id>globalroot-0</id><objectTypeName>GlobalRoot</objectTypeName><name>Global</name></scope></securitygroup>’
// Uncomment the next line to dump the generated string to the debug
//System.debug(“Generated string is: ” + stringbody);
// HTTP POST request is prepared using the URL: /2.0/services/securitygroup/bulk/globalroot-0
// Note that “/api” is not shown in the request URL as it’s already set in the endpointconfiguration
// scopeId is globalroot-0
var request = restHost.createRequest(“POST”, “/2.0/services/securitygroup/bulk/globalroot-0”, stringbody);
request.contentType = “application/xml”;
//Get ObjectID on Securitygroup created.
var request = restHost.createRequest(“GET”, “2.0/services/securitygroup/scope/globalroot-0”); //List all the SecurityGroups
request.contentType = “application/xml”;
System.log(“Check all SecurityGroups- Request URL: ” + request.fullUrl); //Display request URL
var response = request.execute();
var SecGrouplistXML = new XML(response.contentAsString); //Convert response as String
var SecGroupList = SecGrouplistXML.securitygroup;
var SecGroupID = [];
for each (var secgroup in SecGroupList) {
if (secgroup.name.text() == ipSetName) {
if (SecGroupID.length == null) throw(“No SecGroupID found”);
System.log(“SecurityGroup-Id found: “+SecGroupID);
//Add IpSet as Member to SecurityGroup with PUT command against
var put = “/2.0/services/securitygroup/securitygroup-31/members/”
var request = restHost.createRequest(“PUT”, “/2.0/services/securitygroup/”+ SecGroupID +”/members/”+IpSetID);
request.contentType = “application/xml”;

And Last the fourth:

//This Scriptable Task GET:s an XML the ID for the SecurityPolicy called “int30 Security Policy”.
//Then modifies the XML and changes the values for the Securitygroup and the name with the ipSetName and the SecurityGroup ID.
//To update a security policy, one must first fetch it with GET first.
//You then edit the received XML and pass it back as the input with PUT. The specified configuration replaces the current configuration.
//You can retrieve a specific security policy by specifying its ID or all security policies.
//Retrieve the existing configuration of “int30 security Policy”
var request = restHost.createRequest(“GET”, “2.0/services/policy/securitypolicy/policy-6”);
request.contentType = “application/xml”;
//Execute the HTTP request
System.debug(“GET Request URL: ” + request.fullUrl);
var response = request.execute();
var secpolicydocument = XML(response.contentAsString);
System.debug(“SecurityGroupID is now: ” + SecGroupID);
//Change the ObjectID in the SecondarySecurityGroup in the Security Policy to the ID of the SecGroupID
secpolicydocument.actionsByCategory.action.secondarySecurityGroup.objectId = SecGroupID;
secpolicydocument.actionsByCategory.action.secondarySecurityGroup.name = ipSetName;
System.debug(“HTTP Body being sent: ” + xmlbody);
var request = restHost.createRequest(“PUT”, “2.0/services/policy/securitypolicy/policy-6”,xmlbody);
request.contentType = “application/xml”;
var response = request.execute();

Next Up..

Create a XaaS Blueprint with the workflow in vRealize Automation and add it to our Multimachine Blueprint

In this section we want to make sure that the created workflow is run in our Multimachine Blueprint. And make sure that the workflow is run last after the VMs and Edge Loadbalancer has all been created.

First we go to vRA and go to the Design tab and select XaaS Blueprints.

We create a new XaaS Blueprint and name it: Get Edge LB VIP and add to FW SecPolicy Group.

Then we go to our Multimachine Blueprint and Edit it.

In the Blueprint we drag the XaaS on to the canvas. Here it is important to drag the bullet and create an arrow to point to the Loadbalancer. This creates a dependency so that the XaaS blueprint is run after the LB is created. To read mor about dependencies see this linkWe then save the Blueprint and next go to the catalog tab in vRA and run it.If we go into our Request and select Execution Information we see the steps being performed and when all is completed it should show success.

We can also go into our vCenter server and check the NSX Manager and Service Composer to see the modifications and creation of the machines and LB.

First we check the Securitygroup created: We see that the name of the IPSet+EdgeName is a created Securitygroup that includes the IPset.We then check the IP Sets and see that the IP Set is created with the IP of the Edge Loadbalancer VIP.Next we go into the Security Policies tab and select the int30 Security Policy
Here we see the first Firewall rule has been changed and the IPSet-Edge-MultiTierNAT-33db…. Security Group has been added to the Allow LB to RDP to VM rule.

This was it and last we can also check and do an RDP session against the LB VIP and against the VM IP to see the results.The IP address to the VM is and the Edge LB VIP is We first try to RDP to the VM. and see that it don’t allow RDP.Next we try RDP against the LB VIP and it works.
We also do a hostname check on the VM and see that we are in the actual VM behind the Loadbalancer.

The End. Thanx for reading…

VCP6-NV VMware Certified Professional 6 ‚Äď Network Virtualization Exam

Today I passed the VMware Certified Professional 6 ‚Äď Network Virtualization Exam.

Here are my recommendations to pass the exam:
Attend the course NSX: Install, Configure, Manage [V6.2]
Study and follow the Blueprint 
Do Hands on Labs and search the documentation for the labs at: http://docs.hol.vmware.com and filter to NSX.


Create Multimachine Blueprint with vRealize Automation and VMware NSX


1st Virt blog

Welcome to my new Blog.

I will focus on delivering things that I myself am experiencing in my daily work as a Virtualization Consultant in Sweden.

My main interests are working with Vmware products of different kind. I tend to focus on the latest and most challanging products that have a future in todays ever evolving Information age.

I’m a Senior infrastructure consultant and architect with a focus on virtualization and datacenter. Extensive experience in assignments regarding design and technical operation of large datacenter solutions including server, storage and virtualization. I’ve been active in the role of infrastructure architect with the design and implementation of redundant environments based on Windows Server and. Virtualized¬†environments.

At the moment I’m working for a Consultant company in Sweden called Real Time Services. That’s the number one company in Sweden that specializes with VMware.

Load more