Log analysis has always been a standardised practice for activities such as root cause analysis or advanced troubleshooting. However, ingesting and analysing these logs from different devices, types, locations and formats can be a challenge. In this post, we have a look at vRealize Log Insight and what it can deliver.

What is it?

vRealize Log Insight is a product in the vRealize suite specifically designed for heterogeneous and scalable log management across physical, virtual and cloud-based environments. It is designed to be agnostic across what it can ingest logs from and is therefore valid candidate in a lot of deployments.

Additionally, any customer with a vCenter Server Standard or above license is entitled to a free 25 OSI pack. OSI is known as “Operating System Instance” and is broadly defined as a managed entity which is capable of generating logs. For example, a 25 OSI pack license can be used to cover a vCenter server, a number of ESXi hosts and other devices covered either natively or via VMware Content Packs (with the exception of Custom and 3^rd party content packs – standalone vRealize Log Insight is required for this feature).

Current Challenges

Modern datacenters and cloud environments are rarely consumed by homogeneous solutions. Customers use a number of different technologies from different vendors and operating systems. With this comes a number of challenges:

• The inconsistent format of log types – vCenter/ESXi uses syslog for logging, Windows has a bespoke method, applications may simply write data to a file in a specific format. This can require a number of tools/skills to read, interpret and action from this data.
• Silos of information – The decentralised nature of dispersed logging causes this information to be siloed in different areas. This can have an impact on resolution times for incidents and accuracy of root cause analysis.
• Manual analysis – Simply logging information can be helpful, but the reason why this is required is to perform the analysis. In some environments, this is a manual process performed by a systems administrator.
• Not scalable – As environments grow larger and more complex having silos of differentiating logging types and formats becomes unwieldy to manage.
• Cost – Man hours used to perform manual analysis can be costly.
• No Correlation – Siloed logs doesn’t cater for any correlation of events/activities across an environment. This can greatly impede efforts in performing activities such as root cause analysis.

Addressing Challenges With vRealize Log Insight

Below are examples of how vRealize Log Insight can address the aforementioned challenges.

• Create structure from unstructured data – Collected data is automatically analysed and structured for ease of reporting.
• Centralised logging – vRealize Log Insight centrally collates logs from a number of sources which can then be accessed through a single management interface.
• Automatic analysis – Logs are collected in near real-time and alerts can be configured to inform users of potential issues and unexpected events.
• Scalable – Advanced licenses of vRealize Log insight include additional features such as Clustering, High Availability, Event Forwarding and Archiving to facilitate a highly scalable, centralised log management solution. vRealize Log Insight is also designed to analyse massive amounts of log data.
• Cost – Automatic analysis of logs and alerting can assist with reducing man-hours spent manually analysing logs, freeing up IT staff to perform other tasks.
• Log Correlation – Because logs are centralised and structured events across multiple devices/services can be correlated to identify trends and patterns.

Extensibility

vRealize Log Insight’s capabilities can be extended by the use of content packs. Content packs are available from the VMware marketplace (https://marketplace.vmware.com/vsx/?contentType=2)

Content packs are published either by VMware directly or from vendors to support their own devices/solutions. Examples include:

• Apache Web Service
• Brocade Devices
• Cisco Devices
• Dell | EMC Devices
• F5 Devices
• Juniper Devices
• Microsoft Active Directory
• Nimble Devices
• VMware SRM

Closing Thoughts

It’s surprising how underused vRealize Log Insight is considering it comes bundled in as part of any valid vSphere Standard or above license. The modular design of the solution allowing third-party content packs adds a massive degree of flexibility which is not common amongst other centralised logging tools. 

The Software Defined Data Center (SDDC for short) has become a widely adopted and embraced model for modern datacentre implementations. Conveying the benefits of the SDDC, particularly the non-technical aspects can be a challenge. In this blog post we take a practical example of a single activity we can automate in NSX and the benefits that come from it, both technical and non-technical.

The NSX API

An API (Application Programming Interface), in simple terms is an intermediary that allows two applications to communicate with each other via a common syntax. Although we may not be aware of it, it’s likely we use API’s every day when we use applications such as Facebook, LinkedIn, vSphere and countless others. For example, when you create a logical switch in the vSphere web client, behind the scenes an API call is made to the NSX manager to facilitate that request.

The NSX API is based on REST which leverages HTTPS requests to GET, PUT, POST and DELETE data from the NSX ecosystem:

• GET – Retrieve an entity
• PUT – Create an entity
• POST – Update an entity
• DELETE – Remove an entity

An entity can be a variety of NSX objects such as Logical Switches, Distributed Routers, Edge Gateways, Firewall rules, etc.

Options for working with the NSX API

Several avenues exist for working with the rest API, each having their own advantages and disadvantages:

• Direct API calls via REST client – These can be made via clients such as Postman. These calls are static and are therefore suitable for one-off requests.

• PowerNSX – PowerNSX is a PowerShell module that enables the consumption of API calls via Powershell cmdlets. It’s an open source community project but is not supported by VMware. Also, not all API calls are currently exposed as cmdlets.
• API calls via code – API calls can be made from a variety of programming libraries (Powershell, C#, Java, etc) which add functionality by adding an element of dynamic input. We use this as an example in this blog.

Practical example – Creating new networks in a legacy virtualised compute environment

To illustrate the power of automating NSX via automation let’s take an example activity and break it down into respective tasks. In this example, we want to create an N-tier network (IE a network comprising of Web, App and DB tiers which are routable and sit behind a perimeter firewall).

Depending on factors such as the number of vendors used and the structure of the IT team, we can see that executing a relatively simple task of creating an N-Tier routable, secure network for the purposes of consumption could:

• Involve multiple network teams (vSphere admin/network admin/security admin)
• Involve multiple tools (in this example tools from vSphere, Cisco, Juniper and Sonicwall)

This operational complexity can hinder the speed and agility of a business due to factors such as:

• Multiple teams need to collaborate. Collaboration between vSphere / Network / Security teams can be time-consuming
• Multiple tools/skillsets required. In the example above skills pertaining to Sonicwall, Juniper, Cisco and vSphere are required to create a secure network topology

Practical example – Automating in NSX

To demonstrate the automation capabilities designed to address the example a Powershell script was created to facilitate API calls directly to NSX. The advantage of doing this is:

• API calls are supported by VMware.
• The entire API ecosystem is exposed for consumption.
• Powershell can prompt the user for information, which is then used to dynamically populate API requests.
• All tiers of the network are created and managed by a single management plane.

This script starts with the layer 2 logical switches and then moves up the networking stack configuring the layer 3 and perimeter elements of this network:

For each logical network we prompt the user for the following:

• Name – What we want to call the logical network
• Network Range – The intended network range for this network. This is used to determine the DLR’s interface on it
• Network Description – What we provide as the description
• Network Type – Simply put, Uplinks are used for peering (North/South) traffic. We need one uplink network to facilitate the peering between the DLR and ESG

Once the user has put in the required networks, API calls are executed from the Powershell script to create the networks:

Next is to prompt the user for the DLR and ESG names:

This information is used to construct the Distributed Logical Router (DLR) and Edge Services Gateway (ESG) devices via API calls:

At this stage, the following has been created:

At which point the script outputs the total amount of time elapsed to construct this topology in NSX (including the time taken for the user to input the data for).

In this example it took 291.7 seconds (4.9 minutes) to construct the following:

• Create 3 internal logical switches (for VM traffic)
• Create 1 uplink logical switch (for BGP peering)
• Create 1 DLR and configure interfaces on each internal logical switch (default gateway)
• Create 1 ESG and configure interface for BGP peering
• Configure BGP dynamic routing

Not bad at all.

To validate the routing, we can simply log on to the ESG and check its routing table:

We can see the ESG has learnt (by BGP) the networks that reside on our DLR.

This is one of the almost endless examples of exposing and leveraging the NSX API.

For anyone interested in the Powershell script – I intend to upload the code once I’ve added some decent input validation.