# Using Beats to forward host logs to Malcolm Because Malcolm uses components of the open source data analysis platform [Elastic Stack](https://www.elastic.co/elastic-stack), it can accept various host logs sent from [Beats](https://www.elastic.co/beats/#the-beats-family), Elastic Stack's lightweight data shippers. These Beats generally include prebuilt Kibana dashboards for each of their respective data sets. ## Examples Some examples include: * [Auditbeat](https://www.elastic.co/beats/auditbeat) - [`auditd` logs](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-auditd.html) on Linux hosts - [file integrity monitoring](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-file_integrity.html) on Linux, macOS (Darwin) and Windows hosts - [system state](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-system.html) including host, process, login, package, socket and user information on Linux, with some data sets supported on macOS and Windows hosts (apparently not available with the [Open Source Elastic license](https://www.elastic.co/subscriptions)) * [Filebeat](https://www.elastic.co/beats/filebeat) - [system logs](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html) (syslog and authentication logs) on Linux hosts - log output from [many products](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html) across Beats-supported platforms - arbitrary textual [log files](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html) * [Metricbeat](https://www.elastic.co/beats/metricbeat) - [system](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-system.html) resource utilization and process information - metrics from [many products](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html) across Beats-supported platforms * [Packetbeat](https://www.elastic.co/beats/packetbeat) - host-based packet inspection for [many protocols](https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-protocols.html) (supports `libpcap` on Linux, [macOS](https://formulae.brew.sh/formula/libpcap) and [Windows](https://nmap.org/npcap/); and `af_packet` on Linux) * [Winlogbeat](https://www.elastic.co/downloads/beats/winlogbeat) * [Custom](https://www.elastic.co/guide/en/beats/devguide/current/index.html) Beats * [Community-contributed](https://www.elastic.co/guide/en/beats/devguide/current/community-beats.html) Beats ## Convenience configuration scripts and sample configurations Two scripts are provided here for your convenience in configuring and running Beats to forward log data to Malcolm: [beat_config.py](./beat_config.py) and [beat_run.py](./beat_run.py). These Python scripts should run on Linux, macOS and Windows hosts with either Python 2 or Python 3. Sample configurations are also provided for several beats for [Linux](./linux_vm_example) and [Windows](./windows_vm_example) hosts, as well as `Vagrantfile`s for setting up and running [VirtualBox](https://www.virtualbox.org/) VMs under [Vagrant](https://www.vagrantup.com/intro). For further information, downloads, documentation or support for Beats, see the [Beats Platform Reference](https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html) or the [Beats category](https://discuss.elastic.co/c/elastic-stack/beats) on the Elastic forums. ### Example: Windows configuration and run ``` PS C:\Program Files\winlogbeat> dir Directory: C:\Program Files\winlogbeat Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 7/27/2020 8:49 AM kibana d----- 7/27/2020 8:49 AM module -a---- 3/26/2020 5:33 AM 41 .build_hash.txt -a---- 7/27/2020 8:50 AM 25799 beat_common.py -a---- 7/27/2020 8:50 AM 2525 beat_config.py -a---- 7/27/2020 8:50 AM 2244 beat_run.py -a---- 3/26/2020 5:32 AM 163122 fields.yml -a---- 7/27/2020 8:49 AM 878 install-service-winlogbeat.ps1 -a---- 3/26/2020 4:44 AM 13675 LICENSE.txt -a---- 3/26/2020 4:44 AM 328580 NOTICE.txt -a---- 3/26/2020 5:33 AM 825 README.md -a---- 3/26/2020 5:33 AM 254 uninstall-service-winlogbeat.ps1 -a---- 3/26/2020 5:33 AM 47818752 winlogbeat.exe -a---- 3/26/2020 5:32 AM 47900 winlogbeat.reference.yml -a---- 7/27/2020 8:50 AM 1349 winlogbeat.yml PS C:\Program Files\winlogbeat> .\beat_config.py -c .\winlogbeat.yml -b winlogbeat Append connectivity boilerplate to .\winlogbeat.yml? (y/N): y Created winlogbeat keystore Configure winlogbeat Elasticsearch connectivity? (Y/n): y Enter Elasticsearch connection protocol (http or https) [https]: https Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none Enter Elasticsearch connection host: 172.15.0.41:9200 Configure winlogbeat Kibana connectivity? (Y/n): y Enter Kibana connection protocol (http or https) [https]: https Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none Enter Kibana connection host: 172.15.0.41:5601 Configure winlogbeat Kibana dashboards? (Y/n): y Enter directory containing Kibana dashboards [C:\Program Files\winlogbeat\kibana]: C:\Program Files\winlogbeat\kibana Enter HTTP/HTTPS server username: sensor Enter password for sensor: Enter password for sensor (again): Generated keystore for winlogbeat BEAT_KIBANA_SSL_VERIFY BEAT_ES_HOST BEAT_ES_PROTOCOL BEAT_ES_SSL_VERIFY BEAT_KIBANA_HOST BEAT_HTTP_PASSWORD BEAT_HTTP_USERNAME BEAT_KIBANA_DASHBOARDS_ENABLED BEAT_KIBANA_DASHBOARDS_PATH BEAT_KIBANA_PROTOCOL PS C:\Program Files\winlogbeat> .\beat_run.py -c .\winlogbeat.yml -b winlogbeat 2020-07-27T09:00:17.472-0700 INFO instance/beat.go:622 Home path: [C:\Program Files\winlogbeat] Config path: [C:\Program Files\winlogbeat] Data path: [C:\Program Files\winlogbeat] Logs path: [C:\Program Files\winlogbeat\logs] 2020-07-27T09:00:17.474-0700 INFO instance/beat.go:630 Beat ID: c38487f0-ea87-477b-aa93-376eb40949f4 … ^C KeyboardInterrupt 2020-07-27T09:00:24.783-0700 INFO instance/beat.go:445 winlogbeat stopped. ``` ### Example: Linux configuration and run ``` root@vagrant:/opt/filebeat# ls -l total 4 -rw------- 1 root root 431 Jul 27 16:08 filebeat.yml root@vagrant:/opt/filebeat# beat_config.py -c ./filebeat.yml -b filebeat Append connectivity boilerplate to ./filebeat.yml? (y/N): y Create symlink to module path /usr/share/filebeat/module as /opt/filebeat/module? (Y/n): y Created filebeat keystore Configure filebeat Elasticsearch connectivity? (Y/n): y Enter Elasticsearch connection protocol (http or https) [https]: https Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none Enter Elasticsearch connection host: 172.15.0.41:9200 Configure filebeat Kibana connectivity? (Y/n): y Enter Kibana connection protocol (http or https) [https]: https Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none Enter Kibana connection host: 172.15.0.41:5601 Configure filebeat Kibana dashboards? (Y/n): y Enter directory containing Kibana dashboards [/usr/share/filebeat/kibana]: /usr/share/filebeat/kibana Enter HTTP/HTTPS server username: sensor Enter password for sensor: Enter password for sensor (again): Generated keystore for filebeat BEAT_KIBANA_PROTOCOL BEAT_KIBANA_SSL_VERIFY BEAT_ES_PROTOCOL BEAT_ES_SSL_VERIFY BEAT_KIBANA_DASHBOARDS_ENABLED BEAT_KIBANA_DASHBOARDS_PATH BEAT_ES_HOST BEAT_HTTP_PASSWORD BEAT_HTTP_USERNAME BEAT_KIBANA_HOST root@vagrant:/opt/filebeat# beat_run.py -c ./filebeat.yml -b filebeat 2020-07-27T16:12:43.270Z INFO instance/beat.go:622 Home path: [/opt/filebeat] Config path: [/opt/filebeat] Data path: [/opt/filebeat/data] Logs path: [/opt/filebeat/logs] 2020-07-27T16:12:43.270Z INFO instance/beat.go:630 Beat ID: 759019e0-705c-4a16-87a2-52e9a5f6e799 … ^C KeyboardInterrupt 2020-07-27T16:13:10.816Z INFO beater/filebeat.go:443 Stopping filebeat ``` # Copyright [Malcolm](https://github.com/idaholab/Malcolm) is Copyright 2021 Battelle Energy Alliance, LLC, and is developed and released through the cooperation of the Cybersecurity and Infrastructure Security Agency of the U.S. Department of Homeland Security. See [`License.txt`](https://raw.githubusercontent.com/idaholab/Malcolm/master/License.txt) for the terms of its release. ### Contact information of author(s): [Seth Grover](mailto:malcolm.netsec@gmail.com?subject=Malcolm)