added Malcolm
This commit is contained in:
181
Vagrant/resources/malcolm/scripts/beats/README.md
Normal file
181
Vagrant/resources/malcolm/scripts/beats/README.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# Using Beats to forward host logs to Malcolm
|
||||
|
||||
Because Malcolm uses components of the open source data analysis platform [Elastic Stack](https://www.elastic.co/elastic-stack), it can accept various host logs sent from [Beats](https://www.elastic.co/beats/#the-beats-family), Elastic Stack's lightweight data shippers. These Beats generally include prebuilt Kibana dashboards for each of their respective data sets.
|
||||
|
||||
## Examples
|
||||
|
||||
Some examples include:
|
||||
|
||||
* [Auditbeat](https://www.elastic.co/beats/auditbeat)
|
||||
- [`auditd` logs](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-auditd.html) on Linux hosts
|
||||
- [file integrity monitoring](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-file_integrity.html) on Linux, macOS (Darwin) and Windows hosts
|
||||
- [system state](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-system.html) including host, process, login, package, socket and user information on Linux, with some data sets supported on macOS and Windows hosts (apparently not available with the [Open Source Elastic license](https://www.elastic.co/subscriptions))
|
||||
* [Filebeat](https://www.elastic.co/beats/filebeat)
|
||||
- [system logs](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html) (syslog and authentication logs) on Linux hosts
|
||||
- log output from [many products](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html) across Beats-supported platforms
|
||||
- arbitrary textual [log files](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html)
|
||||
* [Metricbeat](https://www.elastic.co/beats/metricbeat)
|
||||
- [system](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-system.html) resource utilization and process information
|
||||
- metrics from [many products](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html) across Beats-supported platforms
|
||||
* [Packetbeat](https://www.elastic.co/beats/packetbeat)
|
||||
- host-based packet inspection for [many protocols](https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-protocols.html) (supports `libpcap` on Linux, [macOS](https://formulae.brew.sh/formula/libpcap) and [Windows](https://nmap.org/npcap/); and `af_packet` on Linux)
|
||||
* [Winlogbeat](https://www.elastic.co/downloads/beats/winlogbeat)
|
||||
* [Custom](https://www.elastic.co/guide/en/beats/devguide/current/index.html) Beats
|
||||
* [Community-contributed](https://www.elastic.co/guide/en/beats/devguide/current/community-beats.html) Beats
|
||||
|
||||
## Convenience configuration scripts and sample configurations
|
||||
|
||||
Two scripts are provided here for your convenience in configuring and running Beats to forward log data to Malcolm: [beat_config.py](./beat_config.py) and [beat_run.py](./beat_run.py). These Python scripts should run on Linux, macOS and Windows hosts with either Python 2 or Python 3.
|
||||
|
||||
Sample configurations are also provided for several beats for [Linux](./linux_vm_example) and [Windows](./windows_vm_example) hosts, as well as `Vagrantfile`s for setting up and running [VirtualBox](https://www.virtualbox.org/) VMs under [Vagrant](https://www.vagrantup.com/intro).
|
||||
|
||||
For further information, downloads, documentation or support for Beats, see the [Beats Platform Reference](https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html) or the [Beats category](https://discuss.elastic.co/c/elastic-stack/beats) on the Elastic forums.
|
||||
|
||||
### Example: Windows configuration and run
|
||||
|
||||
```
|
||||
PS C:\Program Files\winlogbeat> dir
|
||||
|
||||
Directory: C:\Program Files\winlogbeat
|
||||
|
||||
Mode LastWriteTime Length Name
|
||||
---- ------------- ------ ----
|
||||
d----- 7/27/2020 8:49 AM kibana
|
||||
d----- 7/27/2020 8:49 AM module
|
||||
-a---- 3/26/2020 5:33 AM 41 .build_hash.txt
|
||||
-a---- 7/27/2020 8:50 AM 25799 beat_common.py
|
||||
-a---- 7/27/2020 8:50 AM 2525 beat_config.py
|
||||
-a---- 7/27/2020 8:50 AM 2244 beat_run.py
|
||||
-a---- 3/26/2020 5:32 AM 163122 fields.yml
|
||||
-a---- 7/27/2020 8:49 AM 878 install-service-winlogbeat.ps1
|
||||
-a---- 3/26/2020 4:44 AM 13675 LICENSE.txt
|
||||
-a---- 3/26/2020 4:44 AM 328580 NOTICE.txt
|
||||
-a---- 3/26/2020 5:33 AM 825 README.md
|
||||
-a---- 3/26/2020 5:33 AM 254 uninstall-service-winlogbeat.ps1
|
||||
-a---- 3/26/2020 5:33 AM 47818752 winlogbeat.exe
|
||||
-a---- 3/26/2020 5:32 AM 47900 winlogbeat.reference.yml
|
||||
-a---- 7/27/2020 8:50 AM 1349 winlogbeat.yml
|
||||
|
||||
|
||||
PS C:\Program Files\winlogbeat> .\beat_config.py -c .\winlogbeat.yml -b winlogbeat
|
||||
|
||||
Append connectivity boilerplate to .\winlogbeat.yml? (y/N): y
|
||||
|
||||
Created winlogbeat keystore
|
||||
|
||||
Configure winlogbeat Elasticsearch connectivity? (Y/n): y
|
||||
|
||||
Enter Elasticsearch connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Elasticsearch connection host: 172.15.0.41:9200
|
||||
|
||||
Configure winlogbeat Kibana connectivity? (Y/n): y
|
||||
|
||||
Enter Kibana connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Kibana connection host: 172.15.0.41:5601
|
||||
|
||||
Configure winlogbeat Kibana dashboards? (Y/n): y
|
||||
|
||||
Enter directory containing Kibana dashboards [C:\Program Files\winlogbeat\kibana]: C:\Program Files\winlogbeat\kibana
|
||||
|
||||
Enter HTTP/HTTPS server username: sensor
|
||||
Enter password for sensor:
|
||||
Enter password for sensor (again):
|
||||
|
||||
Generated keystore for winlogbeat
|
||||
BEAT_KIBANA_SSL_VERIFY
|
||||
BEAT_ES_HOST
|
||||
BEAT_ES_PROTOCOL
|
||||
BEAT_ES_SSL_VERIFY
|
||||
BEAT_KIBANA_HOST
|
||||
BEAT_HTTP_PASSWORD
|
||||
BEAT_HTTP_USERNAME
|
||||
BEAT_KIBANA_DASHBOARDS_ENABLED
|
||||
BEAT_KIBANA_DASHBOARDS_PATH
|
||||
BEAT_KIBANA_PROTOCOL
|
||||
|
||||
PS C:\Program Files\winlogbeat> .\beat_run.py -c .\winlogbeat.yml -b winlogbeat
|
||||
|
||||
2020-07-27T09:00:17.472-0700 INFO instance/beat.go:622 Home path: [C:\Program Files\winlogbeat] Config path: [C:\Program Files\winlogbeat] Data path: [C:\Program Files\winlogbeat] Logs path: [C:\Program Files\winlogbeat\logs]
|
||||
2020-07-27T09:00:17.474-0700 INFO instance/beat.go:630 Beat ID: c38487f0-ea87-477b-aa93-376eb40949f4
|
||||
…
|
||||
^C
|
||||
KeyboardInterrupt
|
||||
2020-07-27T09:00:24.783-0700 INFO instance/beat.go:445 winlogbeat stopped.
|
||||
```
|
||||
|
||||
### Example: Linux configuration and run
|
||||
|
||||
```
|
||||
root@vagrant:/opt/filebeat# ls -l
|
||||
total 4
|
||||
-rw------- 1 root root 431 Jul 27 16:08 filebeat.yml
|
||||
|
||||
root@vagrant:/opt/filebeat# beat_config.py -c ./filebeat.yml -b filebeat
|
||||
|
||||
Append connectivity boilerplate to ./filebeat.yml? (y/N): y
|
||||
|
||||
Create symlink to module path /usr/share/filebeat/module as /opt/filebeat/module? (Y/n): y
|
||||
|
||||
Created filebeat keystore
|
||||
|
||||
Configure filebeat Elasticsearch connectivity? (Y/n): y
|
||||
|
||||
Enter Elasticsearch connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Elasticsearch connection host: 172.15.0.41:9200
|
||||
|
||||
Configure filebeat Kibana connectivity? (Y/n): y
|
||||
|
||||
Enter Kibana connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Kibana connection host: 172.15.0.41:5601
|
||||
|
||||
Configure filebeat Kibana dashboards? (Y/n): y
|
||||
|
||||
Enter directory containing Kibana dashboards [/usr/share/filebeat/kibana]: /usr/share/filebeat/kibana
|
||||
|
||||
Enter HTTP/HTTPS server username: sensor
|
||||
Enter password for sensor:
|
||||
Enter password for sensor (again):
|
||||
|
||||
Generated keystore for filebeat
|
||||
BEAT_KIBANA_PROTOCOL
|
||||
BEAT_KIBANA_SSL_VERIFY
|
||||
BEAT_ES_PROTOCOL
|
||||
BEAT_ES_SSL_VERIFY
|
||||
BEAT_KIBANA_DASHBOARDS_ENABLED
|
||||
BEAT_KIBANA_DASHBOARDS_PATH
|
||||
BEAT_ES_HOST
|
||||
BEAT_HTTP_PASSWORD
|
||||
BEAT_HTTP_USERNAME
|
||||
BEAT_KIBANA_HOST
|
||||
|
||||
root@vagrant:/opt/filebeat# beat_run.py -c ./filebeat.yml -b filebeat
|
||||
|
||||
2020-07-27T16:12:43.270Z INFO instance/beat.go:622 Home path: [/opt/filebeat] Config path: [/opt/filebeat] Data path: [/opt/filebeat/data] Logs path: [/opt/filebeat/logs]
|
||||
2020-07-27T16:12:43.270Z INFO instance/beat.go:630 Beat ID: 759019e0-705c-4a16-87a2-52e9a5f6e799
|
||||
…
|
||||
^C
|
||||
KeyboardInterrupt
|
||||
2020-07-27T16:13:10.816Z INFO beater/filebeat.go:443 Stopping filebeat
|
||||
```
|
||||
|
||||
# <a name="Footer"></a>Copyright
|
||||
|
||||
[Malcolm](https://github.com/idaholab/Malcolm) is Copyright 2021 Battelle Energy Alliance, LLC, and is developed and released through the cooperation of the Cybersecurity and Infrastructure Security Agency of the U.S. Department of Homeland Security.
|
||||
|
||||
See [`License.txt`](https://raw.githubusercontent.com/idaholab/Malcolm/master/License.txt) for the terms of its release.
|
||||
|
||||
### Contact information of author(s):
|
||||
|
||||
[Seth Grover](mailto:malcolm.netsec@gmail.com?subject=Malcolm)
|
||||
Reference in New Issue
Block a user