Hi!In this post I will detail the process of what is needed inorder to consume network resources from NSX-T Data Center. As of today with vCloud Director 10 and NSX-T 2.5 there are restrictions, requirements …
In this post I want to describe how a backup of the PostgreSQL database for Cloud Director can be setup to be scheduled
In the VMware documentation there is only a description on how one could take a manual backup of the Postgres DB. In normal Operations t can be convenient to have this automated and scheduled. I want to create a daily backup of the database and save that on the NFS Transfer Store inorder to be able to create Image backup of the NFS Server.
To start with I use an old tool called Cron that’s included in the Cloud Director Cell. Cron is running it’s schedules from the directory that is placed at /etc/cron.d/ So in order to create a backup for Postgres we simply create a new file in the location. Called: vcdpostgres_db_backup
In the file we add the following line that creates a backup everyday at 15:00, this can be customized to be once every week or whatever is suitable.
#m h dom mon dow user command
00 15 * * * postgres /opt/vmware/vpostgres/10/bin/pg_dump vcloud > /opt/vmware/vcloud-director/data/transfer/pgdb-backup/$(date +\%F)_vcloud_postgresdbdump.tgz
00 15 * * * root find /opt/vmware/vcloud-director/data/transfer/pgdb-backup/_vcloud -mtime +5 -type f -delete
The command runs as the Postgres user and creates a dump of the vcloud database to the location of /opt/vmware/vcloud-director/data/transfer/pgdb-backup. It finally checks if there are any files that are older than 5 days and deletes them. Be sure to check that the cron job works by tailing /var/log/cron
So last week 17 September 2020, I passed the VMware Certified Master Specialist Cloud Native 2020 exam.
I can agree with some of my colleagues that have also passed this exam that it was not an easy test to pass since this was my second attempt.
For me the reason it was a bit hard is because I have a background within infrastructure, working with VMware products handling Software Defined Data Centers and NSX-T. Understanding a whole new set of tools and ways working as a Developer/DevOps is challenging but a new and very exciting arena for me.
I believe that the future working with VMware products and also OpenSource Communities is the way to go. Not to get stuck with just Legacy OS:es and Applications not created as Cloud Native Apps. But a Mixture of Cloud Native apps and Legacy Applications and Backend Systems all with philosophy that; Lifecycle handling, Tooling, Automation and Orchestration is one of the key elements in surviving in this fast pace-moving Tech World we live in.
So to give som help to you reading this in how you can walk the walk and talk the talk in getting your own VMware Certified Master Specialist Cloud Native Badge below are my links I followed during this chapter that just has begun.
CKA/CKAD First and foremost it is a requirement to pass one of the exams for CKA, Certified Kubernetes Administrator or CKAD Certified Kubernetes Application Developer. I went with CKA since that’s my background.
The test is not asking any questions regarding PKS Enterprise as it says in the description for the exam. So disregard that. I hope VMware will change this because it creates confusion. The Certification get more into asking questions regarding Docker, Docker Build how to crate a file in the correct order etc. A whole bunch of questions on OPA Open Policy Agent with Rego, Pod Security Policies, Conformance Tests with Sonobuy, Monitoring and writing exporters with Prometheus, Backup with Velero, and Log Forwarding with FluentBit.
I would also advise to create a Kubernetes cluster and test all of the mentioned apps, tools and functionality on your own to get to know how and what they do.
So in this post I wanted to describe how a setup with Cloud Director 10.1 with embedded PostgreSQL DB can be set to automatic failover. As described in VMware documentation se Link
So starting with VMware Cloud Director 10.1 the automatic failover functionallity has been added for the roles related to the database that is embedded in the Appliances. So if for some reason the appliance holding the primary DB role of PostgreSQL cluster failed you would prefer that it failover automatically, so you would not have to bother to manually do that. Compared to before the 10.1 release where that was required.
I have a setup of 3 Cloud Director appliances. 1 Primary and the minimal required amount of 2 Standby.
With a browser against one of my cell appliances I check the status of the DB cluster
Starting up Postman Client against the Cloud Director Appliance API I perform a GET command to list all the nodes in my cluster. Below we notice it is set to manual The command to run against the cell appliance api is: https://cell-name:5480/api/1.0.0/nodes
So let’s change the mode to automatic. By running the command according to the API Guide: https://cell-name:5480/api/1.0.0/nodes/failover/automatic
Then with a new GET as before we se that the mode is now set to automatic
Verifying with the browser we see the mode is change in the UI also.
This concludes this post on how to set the mode of the roles to be automatic from manual.
In this short post I wanted to describe a procedure on how you should put you Cloud Director 10.1 appliances with the embedded PostgreSQL DB into maintenance mode. Both for the VCD Service and also the DB to be moved if the cell is a Primary cell in the DB Cluster.
The case of going into Maintenance can be that you need to perform a planned upgrade or decommission a cell. If the appliance cell is holding the Primary PostgreSQL DB role, you also fail the primary role over to a Standby DB Cell and execute the following commands:
On all Cells that are members of the DB Cluster run the below command to put them in Maintenance mode:
Hi! In this post I will detail the process of what is needed inorder to consume network resources from NSX-T Data Center. As of today with vCloud Director 10 and NSX-T 2.5 there are restrictions, requirements and design decisions that are related to both vCloud Director and NSX-T that must be kept in mind before deciding to go NSX-T only. Tomas Fojta has at his blog created a great feature comparison between NSX-V and NSX-T and what kind of functionality that you can get out by choosing NSX-T today. There are some stuff that at the moment from a vCloud Director perspective are not working with NSX-T and that is important to consider when planning and choosing NSX-T with your design. Since NSX-V is the release that has been around for along time and features that exists in that platform are not yet fully functional with NSX-T and vCloud Director. So keep that in mind.
Starting any deployment there is a need to create a design. Below image displays how you could setup NSX-T and vCloud Director with SDDC components with a separate Management and shared Edge and Compute Cluster.
The Edge and Compute cluster is managed by its own vCenter server. It is to this cluster that we will connect the vCloud Director, NSX-T Manager and also place the NSX-T Edge appliances where the T0 and T1 Gateways for providing Tenant N/S traffic, routing functionality and stateful services e.g. Edge Firewall and NAT services. Also Tenant Workloads will reside in this cluster. In the shared Edge and Compute cluster vCloud Director will create the Provider vDC Resource Pool needed to consume the resources that the cluster provides. (CPU, RAM, Storage, NSX-T Resources (Logical Network Segments, Gateways etc.)). Inside of the PvDC there will be Tenant Organizations created and for each Organization there can be one or many Organisation Virtual Datacenters, OvDC. Inorder for the Tenants OvDCs to be able to connect their vAPP and Virtual Machine networks and have the traffic to be able to flow N/S there is a need in NSX-T first to create a T0 Gateway. And to this T0 Gateway is where OvDC Tenants connect their T1 Gateways.
NOTE: The following link to VMware Documentation describes the process that is needed to prepare NSX-T.
I will go through the What to do Next process in this post.
So WHAT TO DO NEXT? After you install vCloud Director, you: – Register the NSX-T Manager – Create a Geneve Network Pool that is Backed by NSX-T transport zone. – Import the T0 Gateway create an External Network and bind it to the pre-created T0 Gateway in vCD – Create an OvDC Edge T1 Gateway and connect it to the External Network – Create an OvDC Routed Network and connect it to the OvDC T1 Gateway – Create a SNAT and DNAT rule for the External IP to the internal Virtual Machine Overlay Segment IP and test ping. – Connect a vAPP Virtual Machine to the OvDC Routed Network
Register the NSX-T Manager
Registering the NSX-T Manager is done by logging into vCloud Director provider portal and going to vSphere Resources.
Create a Geneve Network Pool that is Backed by NSX-T transport zone.
Next we create a Network Pool that is backed by NSX-T Geneve transport zone
Create an OvDC Edge T1 Gateway and connect it to the External Network
We now create an OvDC Edge T1 gateway and connect it to the External Network T0 Gateway. The NSX-T Data Center edge gateway provides a routed organization VDC network with connectivity to external networks and can provide services such as network address translation, and firewall. VMware docs link: Add an NSX-T Data Center Edge Gateway
Create an OvDC Routed Network and connect it to the OvDC T1 Gateway
Now logging in as a Tenant Organization administrator we can see the OvDC and here we can create a routed network and connect it to the OvDC T1 Gateway edge. We may also go to NSX-T Manager UI and check that the T1 Gateway has got the new Segment created and attached. VMware docs link: Add a Routed Organization Virtual Data Center Network
Create a SNAT and DNAT rule for the External IP to the internal Virtual Machine Overlay Segment IP and test ping
In this post I wanted to explain something that is not well documented with Vmware today. The topic is how to create a custom dashboard in NSX-T based on Widgets.
In the NSX-T Manager UI interface there are different monitoring dashboards out of the box that one can view and get information from. These dashboards display details about system status, networking and security and compliance reporting. You will find the dashboards by logging into NSX-T Manager and go to Home-> Monitoring Dashboards.
Here we have some already system defined dashboards:
Status of the NSX Manager cluster and resource (CPU, memory, disk) consumption.
NSX-T fabric, including host and edge transport nodes, transport zones, and compute managers.
NSX-T backups, if configured. It is strongly recommended that you configure scheduled backups that are stored remotely to an SFTP site.
Status of endpoint protection deployment.
Networking & Security
Status of groups and security policies
Status of Tier-0 and Tier-1 gateways.
Status of network segments
Status of the load balancer VMs.
Atatus of VPN, virtual private networks.
Advanced Networking & Security
Status of the load balancer services, load balancer virtual servers, and load balancer server pools
Status of firewall, and shows the number of policies, rules, and exclusions list members.
Status of virtual private networks and the number of IPSec and L2 VPN sessions open
Shows the status of logical switches and logical ports, including both VM and container ports.
Displays information regarding if objects are in compliance with set values.
In the NSX-T REST API Guide there is a section called Management Plane API: Dashboard that contains the information that is needed to create a custom dashboard called VIEW along with a widget configuration. Link to NSX-T 2.5 API Reference Guide
You will need Postman or any other API Client that can GET, POST and PUT information to the NSX-T Manager. Below is my first GET command that lists all the Views that are in place and already created by the system.
By looking in the API Guide you can create the first POST command that we will send to NSX-T Manager to create a new view with some widgets POST https://nsx-t-manager/policy/api/v1/ui-views/
After getting an OK from postman we can look in the NSX-T UI once again and see that we have got a new dashboard in the dropdown list called My Own Custom View
Clicking on it will allow us to see the custom widgets that I choose in my API call to add to the dashboard view.
If you for some reason need to delete a widget from the dashboard you need to do an API call GET to list the widget id for the view and then you can do a DELETE call to delete that widget from the view.
So list all views GET https://nsx-t-manager/policy/api/v1/ui-views/
We see the ID for my custom view is View_7a09f510-4d8f-4132-b371-337408004096 So doing a get call against the view will get more information about the view. GET https://nsx-t-manager/policy/api/v1/ui-views/View_7a09f510-4d8f-4132-b371-337408004096
We can now do a DELETE call and remove the widget configuration for Groups since that widget was of no interest. Note that you will need to add /widgetconfigurations/widget_id after the view_id https://nsx-t-manager/policy/api/v1/ui-views/View_7a09f510-4d8f-4132-b371-337408004096/widgetconfigurations/DonutConfiguration_Groups-Status
Refreshing the NSX-T UI we see the widget is now removed.
This blogpost has been prepared to describe the Micro-Segmentation and security conceptual design planning utilizing VMware vRealize Network Insight, vRNI. It can act as a support for anyone that wishes to know how to think and implement doing microsegmentation in a VMware based environment either with NSX-V or NSX-T. Some of my text that is written out in this post is borrowed and referenced from an official document by VMware: Data Center Security and Networking Assessment
My design is based upon the findings, utilizing the network assessment tool performed by VMware vRealize Network Insight.
You can deploy a Micro-Segmentation security architecture, bearing in mind to:
Deploy firewalls to protect the traffic flowing EastWest (e.g., from server to server). The vast majority of the network traffic in a VMware based SDDC is EastWest based. Unprotected EastWest traffic seriously compromises data center security by allowing threats to easily spread throughout the data center.
Implement a solution that can filter all traffic within the virtualized part of the data center, as well as firewall the traffic between systems on the same Layer 2 segment (VLAN). My analysis showed a vast majority of traffic is VM to VM, and a significant amount is between systems on the same VLAN.
Because of its unique position inside the hypervisor layer, VMware NSX is able to have deep visibility into traffic patterns on the network – even when this traffic flows entirely in the virtualized part of the data center. Combining this intelligence with advanced analytics, vRNI Visibility and Operations Platform provides insight for IT managers, enabling them to make better decisions on what and how to protect critical assets.
The standard approach
to securing data centers has emphasized strong perimeter protection to keep
threats on the outside of the network. However, this model is ineffective for
handling new types of threats – including advanced persistent threats and
coordinated attacks. What’s needed is a better model for data center security:
one that assumes threats can be anywhere and probably are everywhere, then acts
accordingly. Micro Segmentation, powered by VMware NSX, not only adopts such
an approach, but also delivers the operational agility of network
virtualization that is foundational to a modern software defined data center.
Cyber threats today
are coordinated attacks that often include months of reconnaissance,
vulnerability exploits, and “sleeper” malware agents that can lie dormant until
activated by remote control. Despite increasing types of protection at the edge
of data center networks – including advanced firewalls, intrusion prevention
systems, and network based malware detection – attacks are succeeding in
penetrating the perimeter, and breaches continue to occur.
The primary issue is
that once an attack successfully gets past the data center perimeter, there are
few lateral controls to prevent threats from traversing inside the network. The
best way to solve this is to adopt a stricter, micro granular security model
with the ability to tie security to individual workloads and the agility to
provision policies automatically.
VMware NSX is a
network virtualization platform that for the first time makes microsegmentation
economically and operationally feasible. NSX provides the networking and
security foundation for the software defined data center (SDDC), enabling the
three key functions of microsegmentation: isolation, segmentation, and
segmentation with advanced services. Businesses gain key benefits with
Network security inside the data center: flexible security policies aligned to virtual network, VM, OS type, dynamic security tag, and more, for granularity of security down to the virtual NIC
Automated deployment for data center agility: security policies are applied when a VM spins up, are moved when a VM is migrated, and are removed when a VM is de-provisioned – no more stale firewall rules.
Integration with leading networking and security infrastructure: NSX is the platform enabling an ecosystem of partners to integrate – adapting to constantly changing conditions in the data center to provide enhanced security. Best of all, NSX runs on existing data center networking infrastructure.
So I started out by drawing up a conceptual design of the test environment.
Provide a Zero Trust security model using Micro-segmentation around organization’s data center applications. Facilitate only the necessary communications both to the applications and between the components of the applications.
The security framework
is described below:
The blacklist rules at the top will block communication from certain IP addresses from accessing the SDDC environment.
Allow bi-directional communication between the Shared Infrastructure Services and all applications that require access to those services
Deny traffic from one environment (TEST) from communicating to another environment (PROD).
Allow SYSTEM1 Application to communicate with DB Server running on the default ports.
Allow DB Server to communicate with SYSTEM1 Application Server
Allow All Clients to communicate with SYSTEM1 and SYSTEM2 Servers
Block any unknown communications except the actual application traffic to and from the SYSTEM1 application.
Block any unknown communications except the actual application traffic and restrict access to the SYSTEM2 application.
Allow the rest of the traffic until Microsegmentation has been performed in the whole environment, then change to Deny the rest of the traffic.
The goal of the security framework is to deny traffic based on certain criteria, explicitly permit what is required and allow by default until Micro-segmentation has been performed throughout the whole environment. The firewall rules to deny traffic from environment to environment, organization to organization is required. For example, if deny Application to Application rule is missing, an app server from SYSTEM1 can communicate with an application server from SYSTEM2 by hitting the allow all traffic to SYSTEM2 servers rule.
There are different permutation and multiple scenarios to handle so there are many potential firewall rules to be allowed that are not known now. Applications can also be running on non-standard ports. In that case, you can manually open up the firewall rules and deny those that are necessary.
order to be modular and scalable when creating firewall rules, security groups
will be based on NSX security tags on the VMs inside the SDDC and IP Sets will
be created for items outside the datacenter. Firewall rules will then be
applied using these security groups. For each VM, it can be tagged with at
least 3 security tags, with 1 of them in each category.
security tags are classified into 3 categories and each category has a prefix
to identify it:
The names illustrated below are a small subset of the actual names to exemplify the NSX security design.
For the tier category,
a VM can belong to multiple tiers. For example, a VM can be tagged with all 3
security tags in the tier category.
For example, a VM can
have the following tags:
This VM can immediately be identified as a TEST VM belonging to the FG-A Organization and the SYSTEM1 application. Using such classification, you could create your security groups accordingly.
To create microsegmentation for systems outside the datacenters. Creation of IP Sets can be used. IP Sets may contain any combination of individual IP addresses, IP ranges and/or subnets to be used as sources and destinations for firewall rules or as members of Security groups.
Below lists down some
of the security groups:
SG-PROD – Include VMs with a tag that contain ST-PROD
SG-TEST – Include VMs with a tag that contain ST-TEST
SG-FG-A – Include VMs with a tag that contain ST-FG-A
DG-FG-B – Include VMs with a tag that contain ST-FG-B
SG-PROD-INFRA-ALL – Include all Infra VMs that are AD/DNS servers
SG-PROD-INFRA-AD – include IP Set of VMs that are AD/DNS servers
SG-PROD-INFRA-NTP – include IP Set of NTP servers or VMs hosting NTP service
SG-PROD-INFRA-SCOM – include IP Set of SCOM servers or VMs hosting SCOM services
SG-PROD-INFRA-SCCM – include IP Set of SCCM servers or VMs hosting SCCM services
SG-PROD-INFRA-MDM – include IP Set of SNOW servers or VMs hosting SNOW services
SG-PROD-INFRA-RDGW – include IP Set of RDGW servers or VMs hosting RDGW service
SG-PROD-INFRA-FS – include IP Set of FS servers or VMs hosting FS services
SG-TEST-APP-SYSTEM1 – Include VMs that belongs to the application ANTURA
SG-TEST-DB – Include the DB VMs that belongs
SG-KLIENT-ALL – Include the IP Sets for all external clients
SG-WindowsServers – Include VMs whose OS starts with Microsoft Windows Server
SG-LinuxServers – Include VMs whose OS contains CentOS, Red Hat etc
A service is a protocol-port combination, and
a service group is a group of services or other service groups. Below lists down some
of the NSX Service groups and Services that can be created and used in
combination with Security Groups when creating Firewall Rules in NSX:
SERVICE GROUP NAME
SERVICE GROUP CONTAINS
SV-INFRA-FS, SV-INFRA-NTP, SV-INFRA-DNS
SYSTEM1 Analysis and Rule Building
Requirements for SYSTEM1
Allow SYSTEM1 Application to communicate with DB Server running on the default ports.
Allow DB Server to communicate with SYSTEM1 Application Server. Allow Clients to communicate with SYSTEM1 Servers. Block any unknown communications except the actual application traffic to and from the SYSTEM1 application.
To start building firewall rules utilization of vRNI is needed. To ‘Plan Security’ for the VM’s utilize vRNI to start by examining the flows of the SYSTEM1 VM to/from other VMs.
Analysis of flows is done by selecting scope
and segment them accordingly based on entities such as VLAN/VXLAN, Security
Groups, Application, Tier, Folder, Subnet, Cluster, virtual machine (VM), Port,
Security Tag, Security Group, and IPSet. The micro-segmentation dashboard provides
the analysis details with the topology diagram. This dashboard consists of the following
Micro-Segments: This widget provides the diagram for topology
planning. You can select the type of group and flows. Based on your inputs, you
can view the corresponding topology planning diagram.
Traffic Distribution: This widget provides the details of the traffic
distribution in bytes.
Top Ports by Bytes: This widget lists the top 100 ports that record
the highest traffic. The metrics for the flow count and the flow volume are
provided. You can view the flows for a particular port by clicking the count of
flows corresponding to that port.
vRNI displays all flows that are inbound, outbound and bi-directional going to the SYSTEM1 server.
By selecting the SYSTEM1 wedge in the circle it is possible to go in deeper and see the actual flows between the application and other servers and services.
Detailed in this section are the services the SYSTEM1 VM are using, the number of external services that are accessed (49), the number of flows that goes to/from (60) and also the recommended Firewall Rules (14) that can be created to micro-segment the server. vRNI is recommending 14 rules to accommodate micro-segmentation for the SYSTEM1 application.
Option exist to export all the recommended rules as CSV for further processing manually or by automation if needed.
The exported table is listed below for the SYSTEM1 recommended firewall rules.
By continuing the procedure detailed for remaining servers, applications, shared infrastructure services and environments with vRNI, going through each application traffic-flows, exporting the recommended firewall rules, micro-segmentation can be implemented with NSX.
A sample build of firewall rules was conceptually created based on what was gathered during the collection and processing of the data
Firewall Rules The table below shows the firewall rules based on the security framework described based on the requirements above:
When the structure is in order, it
is possible to start building the Security Groups, Security, Tags, Services,
and Service Groups in NSX once that has been implemented. The next step when
creating rules and all needed objects to accomplish micro-segmentation it is
important to go through and check the communications with the servers and applications
and verify they’re all still working correctly per the requirements given.
I would also like to show a table from VMware regarding the Segmentation Strategies. Make sure to start Small and work your way through your environments and systems.
Start with doing MacroSegmentation. Meaning start finding out what Environments can/cannot communicate with other Environments.
When that is completed Setup the MesoSegmentation; Meaning go through what Applications within your Environments can/cannot communicate with other Applications inside the same or outside the Environment.
And Lastly do the MicroSegmentation; Meaning go through what Systems inside the Application can/cannot communicate with other Systems inside the Applications and inside the Environments. Inception thinking is needed 🙂
Also a good idea when drawing out the different Environments, Applications and Systems withing each Application is to Build a Segmentation Flow Chart. With it you will get a picture drawn up on how things are connected and interacted with each other and also makes it much easier to establish what can/cannot communicate with each other.
A microsegmentation approach powered by VMware NSX can address the inadequacy of EastWest security controls that affect most data centers. The vRNI Visibility and Operations software helps to jumpstart the journey to micro segmentation by providing actionable insights into how workloads in a data center communicate and plan the segmentation accordingly.
So the latest patch update for NSX Datacenter for vSphere has been release as of 14 dec 2018.
The latest release finally include even more support in the HTML UI in vSphere.
NSX User Interface
VMware NSX – Functionality Updates for vSphere Client (HTML): The following VMware NSX features are now available through the vSphere Client: Logical Switches, Edge Appliance Management, Edge Services (DHCP, NAT), Edge Certificates, Edge Grouping Objects. For a list of supported functionality, please see VMware NSX for vSphere UI Plug-in Functionality in vSphere Client.
Networking and Edge Services
Static Routes per Edge Service Gateway:increases from 2048 to 10,240 static routes for Quad Large and X-Large Edge Service Gateways.
Also some other issues has been resolved with the latest fix. Please stop by the Release notes page to read about these: Release Notes 6.4.4
This means that we can manage more of NSX features from vSphere. Still there is no functionality to access the Edges firewall, VPN and Routing but in time that will hopefully be release.