The article below describes one of the ways to implement vSphere Log Insight into an environment where a (security departments) syslog server is already available and where this server ideally only receives filtered events like security related events.
Let’s start of by saying that with the introduction of ESXi6, VMware makes it possible to do Log Filtering directly on the syslog service. This however is not recommended as it will make troubleshooting potential future issues impossible.
The overview below shows you a basic setup where Log Insight is used by the operations team and the security departments syslog server only receives filtered events relevant to its purpose. These filtered events can be setup in numerous ways like “ everything, except …” or “ only these specific matches … ”. So by making use of these filters you could easily filter out all vpxa, vMotion, scsi, etc events that are not relevant for the security department but are useful for the operations team.
Devices log towards Log Insight; Log Insight applies filtering; Filtered events get forwarded to the security departments syslog server.
Now the biggest security concern with this setup is the fact that filters can be altered, potentially preventing events from reaching the security departments syslog server. Unfortunately Log Insight has not got a simplified audit trail log yet ( this is on the feature request list, thanks to Raymond de Jong ) but it does have its own internal logging available ( thanks to Leon Scheltema for delivering me information around this )
The Log Insight’s Logs are described here and one of them is the / storage/var/loginsight/runtime.log , which is used to track all run time information related to vRealize Log Insight. So if you take a closer look at this log file you can actually find two, among others, important things there:
Login details to Log Insight; Registration whenever the Event Forwarding Config is changed.With this information you can track if the Event Filters are being changed, which is key information in ensuring that the filters are not tampered with. The only thing you need to do is to get this information passed on towards the security departments syslog server as well.
Now let’s take a closer look at the Log Insight appliance itself as this has its own Log Insight linux agent running which can be validated by running:pgrep liagent
Now this might sound a bit confusing but the Log Insight Appliance itself is based on Linux OS as well and comes with the Log Insight Linux Agent preinstalled.
Configuring this local Log Insight Linux agent gives you the possibility to forward its own internal log files towards a syslog destination, which in this case can be the security departments syslog server.
Data flow then looks like this:
Log Insight sends filtered events from all its connected devices Log Insight sends its own internal log(‘s), which include information about logins and modifications to the Event Forwarding Config.
So by enabling these data flows the security department does not get overloaded with useless events from all devices forwarding to Log Insight, as the operations team can filter out on their request and at the same time the security department does get information on whenever filters on Log Insight have been changed.
To configure the local Linux Agent…………..
Below I will show some sample events within the /storage/var/loginsight/runtime.log regarding login attempts and it’s source:
Local User Login[2016-12-28 11:08:29.954+0000] [“https-jsse-nio-443-exec-6″/x.x.x.x INFO] [com.vmware.loginsight.aaa.local.LocalAndActiveDirectoryAuthenticator] [Successful local user login: Local User: Name= admin ] [2016-12-28 11:08:33.109+0000] [“LogSearchMaster-thread-1567″/x.x.x.x INFO] [com.vmware.loginsight.analytics.distributed.DistributedLogSearcher] [Master returned last result: token: 1cb29ba805a10904, type: AQ, ClientInfo(origin:UI_DASHBOARD, userName: admin , userHost: x.x.x.x ), last-response-time: 78 msec progress: 1.0] AD User Login to Log Insight
[2016-12-28 11:19:59.075+0000] [“https-jsse-nio-443-exec-2″/x.x.x.x INFO] [com.vmware.loginsight.aaa.krb5.KrbAuthenticator] [Attempting Kerberos login: [[ user= test01 ], [ domain= test.local ]]] [2016-12-28 11:19:59.175+0000] [“https-jsse-nio-443-exec-2″/x.x.x.x INFO] [com.vmware.loginsight.aaa.ad.ActiveDirectoryAuthenticator] [Logged in user [test01 ] : [ sam: test01 , domain test.local ] in 103ms] [2016-12-28 11:20:02.480+0000] [“LogSearchMaster-thread-1572″/x.x.x.x INFO] [com.vmware.loginsight.analytics.distributed.DistributedLogSearcher] [Master returned first result: token: 60fff58804cb291c, type: AQ, ClientInfo(origin:UI_DASHBOARD, userName: test01 , userHost: x.x.x.x) , first-response-time: 61 msec progress: 1.0] Whenever the Filters are changed [2016-12-28 10:20:22.466+0000] [“DaemonCommands-thread-1253″/ x.x.x.x INFO] [com.vmware.loginsight.daemon.DaemonCommandsHandler] [ Going to apply configuration change #19, changedSections = EventForwardingConfig ]