Restarting Forwards Splunk Will It Forward Same Data Again

As part of the Cloud Adoption team, I am working with Splunk Cloud (and Splunk Enterprise) customers on a daily basis and I become asked questions quite frequently near how to optimize, and effectively reduce, administration overhead. This becomes peculiarly relevant when I am talking with new or relatively new customers that are expanding from a handful of forwarders, into the 100'southward or 1000's of forwarders. And I always say…. start with a Deployment Server.

For larger customers that accept trained and experienced Splunk Administrators, or have engaged with Professional Services, this is a given and typically already exists in their deployments.

On the other end yet, new Splunk Deject and Splunk Enterprise customers may non have this luxury.

This article is for yous.

I won't get into full details on the how and why this works, but I will outline what configurations are needed and how this will scale based on my field experience, and what our all-time practices outlines. The configurations here are based upon Splunk'southward Professional Services Base Configurations toolset.

Assumptions..

This outlines how to configure a DS to deploy apps on your local network. From an compages point of view, the Cloud Forwarder App contains the configs to ship your data to your Splunk Cloud instance. This could exist interchanged with an App that forwards to on-premise Indexers or an HF/UF Aggregation Tier, but that'south a different discussion…

Let'due south get some terminology out of the way…

Deployment Server (DS) – A Splunk Enterprise instance that acts as a centralized configuration manager. It deploys configuration updates to other instances. Also refers to the overall configuration update facility comprising deployment server, clients, and apps.

Deployment Client – A remotely configured Splunk Enterprise instance. Information technology receives updates from the deployment server. Typically these are Splunk Universal Forwarders or Heavy Forwarders.

Server Class – A deployment configuration category shared by a group of deployment clients. A deployment client can belong to multiple server classes.

Deployment App – A unit of content deployed to the members of one or more than server classes.

Then let's dig in!

First off, nosotros demand a dedicated Splunk Heavy Forwarder (HF/HWF) example that volition be the DS. This case should be configured and already sending its data to your Splunk Cloud instance, and this document assumes this is installed in /opt/splunk .

Hither, a virtual machine is more than sufficient, and preferred. But follow the recommended spec for this : 4 cores x eight gb of RAM and sufficient disk space to handle your deployment apps. (Typically 50gb is more than enough!) Additionally, while not required, a 64bit Linux host is ideal and you lot will get the most mileage out of this.

This server also needs to be placed on the network in such a way that all the hosts can communicate with it. This means that firewalls will need to be opened up for the Splunk Management Port to the DS host (TCP:8089 by default) or multiple DS's deployed.

Additionally, we need our "Apps".

In this article nosotros will deploy the Splunk_TA_nix. "100_demostack_splunkcloud" from our Splunk Deject Stack, and org_deployment_client. (More than on this ane later on!)

These Apps demand to all be placed in the /opt/splunk/etc/deployment-apps/ directory. Once these are place here, they will be visible in the Splunk Web Interface, from the Forwarder Management page.

From hither, we are able to build our Server Classes. To do this, we want to consider our Deployment Topology. In a nutshell, a DS can filter based on hostname, IP accost, or machine type. Then we have a few options for deploying to all of our Clients.

At present we will setup our Server Classes..

First we setup a Server Grade for All Clients. Nosotros are going to phone call this "All_Hosts".

Once we create this, we can add Apps and Clients to the Server Class.

Permit'southward add together our org_deployment_client and 100_demostack_splunkcloud Apps to the All_Hosts serverclass.

And next, we demand to add together Clients. At this point, there are no clients connecting to this DS. Yet, since this class is for all clients, we add a Include whitelist of '*'.

Next, repeat the creation of a serverclass, but with the Splunk_TA_nix add added. For filtering on this, until a client connects, you are non able to filter on machine types. This means you lot demand to filter on car name or IP address until the machine types connect. In this example, I created a filter for a host name of "nix-*, ubuntu*".

Once this is done, your DS is ready and awaiting clients to connect!

Connecting Clients..

Previously I mentioned the "org_deployment_client" app. Let's revisit this at present.

Typically, to configure a client to connect to a DS, we either add it through the CLI (via splunk set deploy-poll servername.mydomain.com:8089) or we edit the deploymentclient.conf file in /opt/splunk/etc/system/local and restart..

That's fine! It works… Just.. information technology is local . Once you put information technology there, you lot accept to manually alter it (or if you're lucky, automate it..) Simply I digress.

From the start, allow'due south make an app that connects to the DS.. Hither's where the "org_deployment_client" comes in to play.

Taken from the Splunk PS Base Configs, here is the template..

[deployment-client]
# Set the phoneHome at the end of the PS engagement
# 10 minutes
# phoneHomeIntervalInSecs = 600

[target-broker:deploymentServer]
# Change the targetUri
targetUri = deploymentserver.splunk.mycompany.com:8089

As yous tin guess, we update the targetUri to indicate to the address and management port of our DS. It's highly recommended to use DNS for this, and not an IP address. And as of half-dozen.3, this tin too be a load balancer.. <finally…woot!! >

At present, the about hard role.. The org_deployment_client app needs to be deployed to all our UFs on install, or later on deployment.. This allows united states the ability in the future to change the targetUri and phoneHomeInternvalInSecs without having to bear upon every forwarder! In that location are many ways to attain this, some use git/mercurial/cvs/ script the commitment of this, some build custom install packages that install this automatically.. Others manually deploy this later on installation.. However yous want to do it, do it!

Back on track.. once this is deployed, we install our clients (with the org_deployment_client.) In this case, I don't accept the apps configured to restart Splunk once they are downloaded from the DS, then a manual restart is required. Afterwards, nosotros can check the Forwarder Management GUI and ostend our hosts and the apps deployed..

From here, we take our hosts sending their data logs to Splunk Cloud. This volition include enabled TA's and modular inputs.

There are "Gotchas"… Please Don't do this!

Hither are a few things to take into consideration, and not to do.

i) Search Head Cluster Members (SHC) – These cannot exist part of a DS, the Deployer Node handles this functionality

ii) Index Cluster Members – These cannot be part of a DS, the Cluster Master Node handles deployment of configurations

3) Using Automation ( Puppet / Chef / Ansible etc) – Exist conscientious when using these in conjunction with DS.. configs can disappear and pause…

four) Test your serverclasses.conf changes in a DEV surroundings!!

5) Standardize on a naming convention for your Server Classes and App names. Here I used org_deployment_client , just for your visitor information technology would exist mycompany_deploymentclient_securelan and mycompany_deploymentclient_dmz1 .

There are a lot of features and functionality available in the Deployment Server that I didn't cover hither. Our Didactics team does a wonderful job of teaching this, and Splunk PS can likewise spend a wonderful corporeality of time going over the dissimilar features of the DS and how to get it to scale. Delight reach out if you want to learn more than!

Additional Reading:
Capacity Planning Transmission for Splunk Enterprise
Updating Splunk Enterprise Instances – Deployment server architecture
Updating Splunk Enterprise Instances – Program a deployment
Updating Splunk Enterprise Instances – Configure deployment clients

Thank you,
Eric Six and Dennis Bourg

----------------------------------------------------
Cheers!
Dennis Bourg

glenncassen.blogspot.com

Source: https://www.splunk.com/en_us/blog/platform/adding-a-deployment-server-forwarder-management-to-a-new-or-existing-splunk-cloud-or-splunk-enterprise-deployment.html

Share this post