added Malcolm
This commit is contained in:
1
Vagrant/resources/malcolm/scripts/auth_setup
Symbolic link
1
Vagrant/resources/malcolm/scripts/auth_setup
Symbolic link
@@ -0,0 +1 @@
|
||||
./control.py
|
||||
3
Vagrant/resources/malcolm/scripts/beats/.gitignore
vendored
Normal file
3
Vagrant/resources/malcolm/scripts/beats/.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
.vagrant
|
||||
data
|
||||
logs
|
||||
181
Vagrant/resources/malcolm/scripts/beats/README.md
Normal file
181
Vagrant/resources/malcolm/scripts/beats/README.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# Using Beats to forward host logs to Malcolm
|
||||
|
||||
Because Malcolm uses components of the open source data analysis platform [Elastic Stack](https://www.elastic.co/elastic-stack), it can accept various host logs sent from [Beats](https://www.elastic.co/beats/#the-beats-family), Elastic Stack's lightweight data shippers. These Beats generally include prebuilt Kibana dashboards for each of their respective data sets.
|
||||
|
||||
## Examples
|
||||
|
||||
Some examples include:
|
||||
|
||||
* [Auditbeat](https://www.elastic.co/beats/auditbeat)
|
||||
- [`auditd` logs](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-auditd.html) on Linux hosts
|
||||
- [file integrity monitoring](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-file_integrity.html) on Linux, macOS (Darwin) and Windows hosts
|
||||
- [system state](https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-system.html) including host, process, login, package, socket and user information on Linux, with some data sets supported on macOS and Windows hosts (apparently not available with the [Open Source Elastic license](https://www.elastic.co/subscriptions))
|
||||
* [Filebeat](https://www.elastic.co/beats/filebeat)
|
||||
- [system logs](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html) (syslog and authentication logs) on Linux hosts
|
||||
- log output from [many products](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html) across Beats-supported platforms
|
||||
- arbitrary textual [log files](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html)
|
||||
* [Metricbeat](https://www.elastic.co/beats/metricbeat)
|
||||
- [system](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-system.html) resource utilization and process information
|
||||
- metrics from [many products](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html) across Beats-supported platforms
|
||||
* [Packetbeat](https://www.elastic.co/beats/packetbeat)
|
||||
- host-based packet inspection for [many protocols](https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-protocols.html) (supports `libpcap` on Linux, [macOS](https://formulae.brew.sh/formula/libpcap) and [Windows](https://nmap.org/npcap/); and `af_packet` on Linux)
|
||||
* [Winlogbeat](https://www.elastic.co/downloads/beats/winlogbeat)
|
||||
* [Custom](https://www.elastic.co/guide/en/beats/devguide/current/index.html) Beats
|
||||
* [Community-contributed](https://www.elastic.co/guide/en/beats/devguide/current/community-beats.html) Beats
|
||||
|
||||
## Convenience configuration scripts and sample configurations
|
||||
|
||||
Two scripts are provided here for your convenience in configuring and running Beats to forward log data to Malcolm: [beat_config.py](./beat_config.py) and [beat_run.py](./beat_run.py). These Python scripts should run on Linux, macOS and Windows hosts with either Python 2 or Python 3.
|
||||
|
||||
Sample configurations are also provided for several beats for [Linux](./linux_vm_example) and [Windows](./windows_vm_example) hosts, as well as `Vagrantfile`s for setting up and running [VirtualBox](https://www.virtualbox.org/) VMs under [Vagrant](https://www.vagrantup.com/intro).
|
||||
|
||||
For further information, downloads, documentation or support for Beats, see the [Beats Platform Reference](https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html) or the [Beats category](https://discuss.elastic.co/c/elastic-stack/beats) on the Elastic forums.
|
||||
|
||||
### Example: Windows configuration and run
|
||||
|
||||
```
|
||||
PS C:\Program Files\winlogbeat> dir
|
||||
|
||||
Directory: C:\Program Files\winlogbeat
|
||||
|
||||
Mode LastWriteTime Length Name
|
||||
---- ------------- ------ ----
|
||||
d----- 7/27/2020 8:49 AM kibana
|
||||
d----- 7/27/2020 8:49 AM module
|
||||
-a---- 3/26/2020 5:33 AM 41 .build_hash.txt
|
||||
-a---- 7/27/2020 8:50 AM 25799 beat_common.py
|
||||
-a---- 7/27/2020 8:50 AM 2525 beat_config.py
|
||||
-a---- 7/27/2020 8:50 AM 2244 beat_run.py
|
||||
-a---- 3/26/2020 5:32 AM 163122 fields.yml
|
||||
-a---- 7/27/2020 8:49 AM 878 install-service-winlogbeat.ps1
|
||||
-a---- 3/26/2020 4:44 AM 13675 LICENSE.txt
|
||||
-a---- 3/26/2020 4:44 AM 328580 NOTICE.txt
|
||||
-a---- 3/26/2020 5:33 AM 825 README.md
|
||||
-a---- 3/26/2020 5:33 AM 254 uninstall-service-winlogbeat.ps1
|
||||
-a---- 3/26/2020 5:33 AM 47818752 winlogbeat.exe
|
||||
-a---- 3/26/2020 5:32 AM 47900 winlogbeat.reference.yml
|
||||
-a---- 7/27/2020 8:50 AM 1349 winlogbeat.yml
|
||||
|
||||
|
||||
PS C:\Program Files\winlogbeat> .\beat_config.py -c .\winlogbeat.yml -b winlogbeat
|
||||
|
||||
Append connectivity boilerplate to .\winlogbeat.yml? (y/N): y
|
||||
|
||||
Created winlogbeat keystore
|
||||
|
||||
Configure winlogbeat Elasticsearch connectivity? (Y/n): y
|
||||
|
||||
Enter Elasticsearch connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Elasticsearch connection host: 172.15.0.41:9200
|
||||
|
||||
Configure winlogbeat Kibana connectivity? (Y/n): y
|
||||
|
||||
Enter Kibana connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Kibana connection host: 172.15.0.41:5601
|
||||
|
||||
Configure winlogbeat Kibana dashboards? (Y/n): y
|
||||
|
||||
Enter directory containing Kibana dashboards [C:\Program Files\winlogbeat\kibana]: C:\Program Files\winlogbeat\kibana
|
||||
|
||||
Enter HTTP/HTTPS server username: sensor
|
||||
Enter password for sensor:
|
||||
Enter password for sensor (again):
|
||||
|
||||
Generated keystore for winlogbeat
|
||||
BEAT_KIBANA_SSL_VERIFY
|
||||
BEAT_ES_HOST
|
||||
BEAT_ES_PROTOCOL
|
||||
BEAT_ES_SSL_VERIFY
|
||||
BEAT_KIBANA_HOST
|
||||
BEAT_HTTP_PASSWORD
|
||||
BEAT_HTTP_USERNAME
|
||||
BEAT_KIBANA_DASHBOARDS_ENABLED
|
||||
BEAT_KIBANA_DASHBOARDS_PATH
|
||||
BEAT_KIBANA_PROTOCOL
|
||||
|
||||
PS C:\Program Files\winlogbeat> .\beat_run.py -c .\winlogbeat.yml -b winlogbeat
|
||||
|
||||
2020-07-27T09:00:17.472-0700 INFO instance/beat.go:622 Home path: [C:\Program Files\winlogbeat] Config path: [C:\Program Files\winlogbeat] Data path: [C:\Program Files\winlogbeat] Logs path: [C:\Program Files\winlogbeat\logs]
|
||||
2020-07-27T09:00:17.474-0700 INFO instance/beat.go:630 Beat ID: c38487f0-ea87-477b-aa93-376eb40949f4
|
||||
…
|
||||
^C
|
||||
KeyboardInterrupt
|
||||
2020-07-27T09:00:24.783-0700 INFO instance/beat.go:445 winlogbeat stopped.
|
||||
```
|
||||
|
||||
### Example: Linux configuration and run
|
||||
|
||||
```
|
||||
root@vagrant:/opt/filebeat# ls -l
|
||||
total 4
|
||||
-rw------- 1 root root 431 Jul 27 16:08 filebeat.yml
|
||||
|
||||
root@vagrant:/opt/filebeat# beat_config.py -c ./filebeat.yml -b filebeat
|
||||
|
||||
Append connectivity boilerplate to ./filebeat.yml? (y/N): y
|
||||
|
||||
Create symlink to module path /usr/share/filebeat/module as /opt/filebeat/module? (Y/n): y
|
||||
|
||||
Created filebeat keystore
|
||||
|
||||
Configure filebeat Elasticsearch connectivity? (Y/n): y
|
||||
|
||||
Enter Elasticsearch connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Elasticsearch connection host: 172.15.0.41:9200
|
||||
|
||||
Configure filebeat Kibana connectivity? (Y/n): y
|
||||
|
||||
Enter Kibana connection protocol (http or https) [https]: https
|
||||
|
||||
Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none
|
||||
|
||||
Enter Kibana connection host: 172.15.0.41:5601
|
||||
|
||||
Configure filebeat Kibana dashboards? (Y/n): y
|
||||
|
||||
Enter directory containing Kibana dashboards [/usr/share/filebeat/kibana]: /usr/share/filebeat/kibana
|
||||
|
||||
Enter HTTP/HTTPS server username: sensor
|
||||
Enter password for sensor:
|
||||
Enter password for sensor (again):
|
||||
|
||||
Generated keystore for filebeat
|
||||
BEAT_KIBANA_PROTOCOL
|
||||
BEAT_KIBANA_SSL_VERIFY
|
||||
BEAT_ES_PROTOCOL
|
||||
BEAT_ES_SSL_VERIFY
|
||||
BEAT_KIBANA_DASHBOARDS_ENABLED
|
||||
BEAT_KIBANA_DASHBOARDS_PATH
|
||||
BEAT_ES_HOST
|
||||
BEAT_HTTP_PASSWORD
|
||||
BEAT_HTTP_USERNAME
|
||||
BEAT_KIBANA_HOST
|
||||
|
||||
root@vagrant:/opt/filebeat# beat_run.py -c ./filebeat.yml -b filebeat
|
||||
|
||||
2020-07-27T16:12:43.270Z INFO instance/beat.go:622 Home path: [/opt/filebeat] Config path: [/opt/filebeat] Data path: [/opt/filebeat/data] Logs path: [/opt/filebeat/logs]
|
||||
2020-07-27T16:12:43.270Z INFO instance/beat.go:630 Beat ID: 759019e0-705c-4a16-87a2-52e9a5f6e799
|
||||
…
|
||||
^C
|
||||
KeyboardInterrupt
|
||||
2020-07-27T16:13:10.816Z INFO beater/filebeat.go:443 Stopping filebeat
|
||||
```
|
||||
|
||||
# <a name="Footer"></a>Copyright
|
||||
|
||||
[Malcolm](https://github.com/idaholab/Malcolm) is Copyright 2021 Battelle Energy Alliance, LLC, and is developed and released through the cooperation of the Cybersecurity and Infrastructure Security Agency of the U.S. Department of Homeland Security.
|
||||
|
||||
See [`License.txt`](https://raw.githubusercontent.com/idaholab/Malcolm/master/License.txt) for the terms of its release.
|
||||
|
||||
### Contact information of author(s):
|
||||
|
||||
[Seth Grover](mailto:malcolm.netsec@gmail.com?subject=Malcolm)
|
||||
622
Vagrant/resources/malcolm/scripts/beats/beat_common.py
Normal file
622
Vagrant/resources/malcolm/scripts/beats/beat_common.py
Normal file
@@ -0,0 +1,622 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import getpass
|
||||
import inspect
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
from collections import defaultdict
|
||||
|
||||
from subprocess import (PIPE, STDOUT, Popen, CalledProcessError)
|
||||
|
||||
###################################################################################################
|
||||
ScriptPath = os.path.dirname(os.path.realpath(__file__))
|
||||
|
||||
###################################################################################################
|
||||
# python 2/3 portability
|
||||
PY3 = (sys.version_info.major >= 3)
|
||||
|
||||
# bind raw_input to input in older versions of python
|
||||
try:
|
||||
input = raw_input
|
||||
except NameError:
|
||||
pass
|
||||
|
||||
try:
|
||||
FileNotFoundError
|
||||
except NameError:
|
||||
FileNotFoundError = IOError
|
||||
|
||||
###################################################################################################
|
||||
PLATFORM_WINDOWS = "Windows"
|
||||
PLATFORM_MAC = "Darwin"
|
||||
PLATFORM_LINUX = "Linux"
|
||||
PLATFORM_LINUX_CENTOS = 'centos'
|
||||
PLATFORM_LINUX_DEBIAN = 'debian'
|
||||
PLATFORM_LINUX_FEDORA = 'fedora'
|
||||
PLATFORM_LINUX_UBUNTU = 'ubuntu'
|
||||
|
||||
OPERATION_RUN = 'run'
|
||||
OPERATION_CONFIGURE = 'config'
|
||||
|
||||
BEAT_ES_HOST = "BEAT_ES_HOST"
|
||||
BEAT_ES_PROTOCOL = "BEAT_ES_PROTOCOL"
|
||||
BEAT_ES_SSL_VERIFY = "BEAT_ES_SSL_VERIFY"
|
||||
BEAT_HTTP_PASSWORD = "BEAT_HTTP_PASSWORD"
|
||||
BEAT_HTTP_USERNAME = "BEAT_HTTP_USERNAME"
|
||||
BEAT_KIBANA_DASHBOARDS_ENABLED = "BEAT_KIBANA_DASHBOARDS_ENABLED"
|
||||
BEAT_KIBANA_DASHBOARDS_PATH = "BEAT_KIBANA_DASHBOARDS_PATH"
|
||||
BEAT_KIBANA_HOST = "BEAT_KIBANA_HOST"
|
||||
BEAT_KIBANA_PROTOCOL = "BEAT_KIBANA_PROTOCOL"
|
||||
BEAT_KIBANA_SSL_VERIFY = "BEAT_KIBANA_SSL_VERIFY"
|
||||
|
||||
BEAT_YML_TEMPLATE = """
|
||||
#================================ General ======================================
|
||||
fields_under_root: true
|
||||
|
||||
#================================ Outputs ======================================
|
||||
|
||||
#-------------------------- Elasticsearch output -------------------------------
|
||||
output.elasticsearch:
|
||||
enabled: true
|
||||
hosts: ["${BEAT_ES_HOST}"]
|
||||
protocol: "${BEAT_ES_PROTOCOL}"
|
||||
username: "${BEAT_HTTP_USERNAME}"
|
||||
password: "${BEAT_HTTP_PASSWORD}"
|
||||
ssl.verification_mode: "${BEAT_ES_SSL_VERIFY}"
|
||||
|
||||
setup.template.enabled: true
|
||||
setup.template.overwrite: false
|
||||
setup.template.settings:
|
||||
index.number_of_shards: 1
|
||||
index.number_of_replicas: 0
|
||||
|
||||
#============================== Dashboards =====================================
|
||||
setup.dashboards.enabled: "${BEAT_KIBANA_DASHBOARDS_ENABLED}"
|
||||
setup.dashboards.directory: "${BEAT_KIBANA_DASHBOARDS_PATH}"
|
||||
|
||||
#============================== Kibana =====================================
|
||||
setup.kibana:
|
||||
host: "${BEAT_KIBANA_HOST}"
|
||||
protocol: "${BEAT_KIBANA_PROTOCOL}"
|
||||
username: "${BEAT_HTTP_USERNAME}"
|
||||
password: "${BEAT_HTTP_PASSWORD}"
|
||||
ssl.verification_mode: "${BEAT_KIBANA_SSL_VERIFY}"
|
||||
|
||||
#================================ Logging ======================================
|
||||
logging.metrics.enabled: false
|
||||
"""
|
||||
|
||||
###################################################################################################
|
||||
# print to stderr
|
||||
def eprint(*args, **kwargs):
|
||||
print(*args, file=sys.stderr, **kwargs)
|
||||
|
||||
###################################################################################################
|
||||
# get interactive user response to Y/N question
|
||||
def YesOrNo(question, default=None, forceInteraction=False, acceptDefault=False):
|
||||
|
||||
if default == True:
|
||||
questionStr = "\n{} (Y/n): ".format(question)
|
||||
elif default == False:
|
||||
questionStr = "\n{} (y/N): ".format(question)
|
||||
else:
|
||||
questionStr = "\n{} (y/n): ".format(question)
|
||||
|
||||
if acceptDefault and (default is not None) and (not forceInteraction):
|
||||
reply = ''
|
||||
else:
|
||||
while True:
|
||||
reply = str(input(questionStr)).lower().strip()
|
||||
if (len(reply) > 0) or (default is not None):
|
||||
break
|
||||
|
||||
if (len(reply) == 0):
|
||||
reply = 'y' if default else 'n'
|
||||
|
||||
if reply[0] == 'y':
|
||||
return True
|
||||
elif reply[0] == 'n':
|
||||
return False
|
||||
else:
|
||||
return YesOrNo(question, default=default)
|
||||
|
||||
###################################################################################################
|
||||
# get interactive user response
|
||||
def AskForString(question, default=None, forceInteraction=False, acceptDefault=False):
|
||||
|
||||
if acceptDefault and (default is not None) and (not forceInteraction):
|
||||
reply = default
|
||||
else:
|
||||
reply = str(input('\n{}: '.format(question))).strip()
|
||||
|
||||
return reply
|
||||
|
||||
###################################################################################################
|
||||
# get interactive password (without echoing)
|
||||
def AskForPassword(prompt):
|
||||
reply = getpass.getpass(prompt=prompt)
|
||||
return reply
|
||||
|
||||
###################################################################################################
|
||||
# convenient boolean argument parsing
|
||||
def str2bool(v):
|
||||
if v.lower() in ('yes', 'true', 't', 'y', '1'):
|
||||
return True
|
||||
elif v.lower() in ('no', 'false', 'f', 'n', '0'):
|
||||
return False
|
||||
else:
|
||||
raise ValueError('Boolean value expected')
|
||||
|
||||
###################################################################################################
|
||||
# determine if a program/script exists and is executable in the system path
|
||||
def Which(cmd, debug=False):
|
||||
result = any(os.access(os.path.join(path, cmd), os.X_OK) for path in os.environ["PATH"].split(os.pathsep))
|
||||
if (not result) and (platform.system() == PLATFORM_WINDOWS):
|
||||
result = os.access(os.path.join(os.getcwd(), cmd), os.X_OK)
|
||||
if debug:
|
||||
eprint("Which {} returned {}".format(cmd, result))
|
||||
return result
|
||||
|
||||
###################################################################################################
|
||||
# run command with arguments and return its exit code, stdout, and stderr
|
||||
def check_output_input(*popenargs, **kwargs):
|
||||
|
||||
if 'stdout' in kwargs:
|
||||
raise ValueError('stdout argument not allowed, it will be overridden')
|
||||
|
||||
if 'stderr' in kwargs:
|
||||
raise ValueError('stderr argument not allowed, it will be overridden')
|
||||
|
||||
if 'input' in kwargs and kwargs['input']:
|
||||
if 'stdin' in kwargs:
|
||||
raise ValueError('stdin and input arguments may not both be used')
|
||||
inputdata = kwargs['input']
|
||||
kwargs['stdin'] = PIPE
|
||||
else:
|
||||
inputdata = None
|
||||
kwargs.pop('input', None)
|
||||
|
||||
process = Popen(*popenargs, stdout=PIPE, stderr=PIPE, **kwargs)
|
||||
try:
|
||||
output, errput = process.communicate(inputdata)
|
||||
except:
|
||||
process.kill()
|
||||
process.wait()
|
||||
raise
|
||||
|
||||
retcode = process.poll()
|
||||
|
||||
return retcode, output, errput
|
||||
|
||||
###################################################################################################
|
||||
# run command with arguments and return its exit code, stdout, and stderr
|
||||
def run_process(command, stdout=True, stderr=True, stdin=None, retry=0, retrySleepSec=5, cwd=None, env=None, debug=False):
|
||||
|
||||
retcode = -1
|
||||
output = []
|
||||
|
||||
try:
|
||||
# run the command
|
||||
retcode, cmdout, cmderr = check_output_input(command, input=stdin.encode() if (PY3 and stdin) else stdin, cwd=cwd, env=env)
|
||||
|
||||
# split the output on newlines to return a list
|
||||
if PY3:
|
||||
if stderr and (len(cmderr) > 0): output.extend(cmderr.decode(sys.getdefaultencoding()).split('\n'))
|
||||
if stdout and (len(cmdout) > 0): output.extend(cmdout.decode(sys.getdefaultencoding()).split('\n'))
|
||||
else:
|
||||
if stderr and (len(cmderr) > 0): output.extend(cmderr.split('\n'))
|
||||
if stdout and (len(cmdout) > 0): output.extend(cmdout.split('\n'))
|
||||
|
||||
except (FileNotFoundError, OSError, IOError) as e:
|
||||
if stderr:
|
||||
output.append("Command {} not found or unable to execute".format(command))
|
||||
|
||||
if debug:
|
||||
eprint("{}{} returned {}: {}".format(command, "({})".format(stdin[:80] + bool(stdin[80:]) * '...' if stdin else ""), retcode, output))
|
||||
|
||||
if (retcode != 0) and retry and (retry > 0):
|
||||
# sleep then retry
|
||||
time.sleep(retrySleepSec)
|
||||
return run_process(command, stdout, stderr, stdin, retry-1, retrySleepSec, cwd, env, debug)
|
||||
else:
|
||||
return retcode, output
|
||||
|
||||
###################################################################################################
|
||||
class Beatbox(object):
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def __init__(self, debug=False, ymlFileSpec=None, beatName=None, acceptDefaults=False):
|
||||
self.debug = debug
|
||||
self.acceptDefaults = acceptDefaults
|
||||
self.platform = platform.system()
|
||||
self.ymlFileSpec = ymlFileSpec
|
||||
self.ymlFilePath = os.path.dirname(ymlFileSpec)
|
||||
self.beatName = beatName
|
||||
self.beatExe = beatName
|
||||
self.beatInstallDir = None
|
||||
self.defaultKibanaDashboardDir = None
|
||||
self.keystoreItems = defaultdict(str)
|
||||
for initItem in [BEAT_ES_HOST,
|
||||
BEAT_ES_PROTOCOL,
|
||||
BEAT_ES_SSL_VERIFY,
|
||||
BEAT_HTTP_PASSWORD,
|
||||
BEAT_HTTP_USERNAME,
|
||||
BEAT_KIBANA_DASHBOARDS_ENABLED,
|
||||
BEAT_KIBANA_DASHBOARDS_PATH,
|
||||
BEAT_KIBANA_HOST,
|
||||
BEAT_KIBANA_PROTOCOL,
|
||||
BEAT_KIBANA_SSL_VERIFY]:
|
||||
self.keystoreItems[initItem] = ''
|
||||
self.keystorePath = None
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def __del__(self):
|
||||
# nothing for now
|
||||
pass
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def run_process(self, command, stdout=True, stderr=True, stdin=None, retry=0, retrySleepSec=5):
|
||||
return run_process(command, stdout=stdout, stderr=stderr, stdin=stdin, retry=retry, retrySleepSec=retrySleepSec, debug=self.debug)
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def build_beat_command(self, command):
|
||||
if not Which(self.beatExe, debug=self.debug):
|
||||
raise Exception("Beat executable {} does not exist".format(self.beatExe))
|
||||
|
||||
if not os.path.isfile(self.ymlFileSpec):
|
||||
raise Exception("Beat configuration {} does not exist".format(self.ymlFileSpec))
|
||||
|
||||
# convert paths to absolutes
|
||||
ymlFileSpec = os.path.abspath(self.ymlFileSpec)
|
||||
ymlFilePath = os.path.dirname(ymlFileSpec)
|
||||
|
||||
beatCmd = [self.beatExe, '--path.home', ymlFilePath, '--path.config', ymlFilePath, '--path.data', ymlFilePath if (self.platform == PLATFORM_WINDOWS) else os.path.join(ymlFilePath, 'data'), '--path.logs', os.path.join(ymlFilePath, 'logs'), '-c', ymlFileSpec, '-E', "keystore.path='{}'".format(self.keystorePath)]
|
||||
|
||||
return beatCmd + command if isinstance(command, list) else beatCmd + [ command ]
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def run_beat_command(self, command, stdout=True, stderr=True, stdin=None, retry=0, retrySleepSec=5):
|
||||
return self.run_process(self.build_beat_command(command), stdout=stdout, stderr=stderr, stdin=stdin, retry=retry, retrySleepSec=retrySleepSec)
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_beat_yml(self):
|
||||
if self.debug:
|
||||
eprint("{}: {}".format(self.__class__.__name__, inspect.currentframe().f_code.co_name))
|
||||
|
||||
if (self.ymlFileSpec is not None):
|
||||
|
||||
if os.path.isfile(self.ymlFileSpec):
|
||||
# if it doesn't look like connectivity stuff (at last BEAT_ES_PROTOCOL) is in the YML file, offer to append it
|
||||
if ((len(list(filter(lambda x: BEAT_ES_PROTOCOL in x, [line.rstrip('\n') for line in open(self.ymlFileSpec)]))) == 0) and
|
||||
YesOrNo("Append connectivity boilerplate to {}?".format(self.ymlFileSpec), default=False, acceptDefault=self.acceptDefaults)):
|
||||
with open(self.ymlFileSpec, 'a') as ymlFile:
|
||||
ymlFile.write(BEAT_YML_TEMPLATE)
|
||||
|
||||
else:
|
||||
# generate a boilerplate spec file (output configured, no modules) if the YML file doesn't exist
|
||||
with open(self.ymlFileSpec, 'w') as ymlFile:
|
||||
ymlFile.write(BEAT_YML_TEMPLATE)
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_keystore(self):
|
||||
if self.debug:
|
||||
eprint("{}: {}".format(self.__class__.__name__, inspect.currentframe().f_code.co_name))
|
||||
|
||||
# check if keystore already exists
|
||||
err, out = self.run_beat_command(['keystore', 'list'])
|
||||
if (err == 0) and (len(out) > 0):
|
||||
if not YesOrNo("{} keystore already exists, overwrite?".format(self.beatName), default=False, acceptDefault=self.acceptDefaults):
|
||||
raise Exception("Configuration cancelled by user")
|
||||
|
||||
# create keystore
|
||||
err, out = self.run_beat_command(['keystore', 'create', '--force'])
|
||||
if (err == 0):
|
||||
eprint('\n'.join(out))
|
||||
else:
|
||||
raise Exception("Keystore creation failed: {}".format(out))
|
||||
|
||||
# prompt for and store configuration items
|
||||
for destination in ['Elasticsearch', 'Kibana']:
|
||||
|
||||
if YesOrNo("Configure {} {} connectivity?".format(self.beatName, destination), default=True, acceptDefault=self.acceptDefaults):
|
||||
|
||||
# protocol
|
||||
tmpVal, tmpDefault = '', 'https'
|
||||
while tmpVal not in ['http', 'https']:
|
||||
tmpVal = AskForString("Enter {} connection protocol (http or https) [{}]".format(destination, tmpDefault), default=tmpDefault, acceptDefault=self.acceptDefaults).lower()
|
||||
if (len(tmpVal) == 0): tmpVal = tmpDefault
|
||||
self.keystoreItems[BEAT_ES_PROTOCOL.replace('_ES_', '_KIBANA_' if (destination == 'Kibana') else '_ES_')] = tmpVal
|
||||
|
||||
# SSL verification
|
||||
tmpVal, tmpDefault = '', 'none'
|
||||
while tmpVal not in ['none', 'full']:
|
||||
tmpVal = AskForString("Enter {} SSL verification (none (for self-signed certificates) or full) [{}]".format(destination, tmpDefault), default=tmpDefault, acceptDefault=self.acceptDefaults).lower()
|
||||
if (len(tmpVal) == 0): tmpVal = tmpDefault
|
||||
self.keystoreItems[BEAT_ES_SSL_VERIFY.replace('_ES_', '_KIBANA_' if (destination == 'Kibana') else '_ES_')] = tmpVal
|
||||
|
||||
# host
|
||||
tmpVal, tmpDefault = '', ''
|
||||
while (len(tmpVal) == 0):
|
||||
tmpVal = AskForString("Enter {} connection host".format(destination), default=tmpDefault, acceptDefault=self.acceptDefaults)
|
||||
self.keystoreItems[BEAT_ES_HOST.replace('_ES_', '_KIBANA_' if (destination == 'Kibana') else '_ES_')] = tmpVal
|
||||
|
||||
if (BEAT_KIBANA_HOST in self.keystoreItems):
|
||||
|
||||
# configure kibana dashboards
|
||||
if YesOrNo("Configure {} Kibana dashboards?".format(self.beatName), default=True, acceptDefault=self.acceptDefaults):
|
||||
self.keystoreItems[BEAT_KIBANA_DASHBOARDS_ENABLED] = 'true'
|
||||
|
||||
# kibana dashboards
|
||||
tmpVal, tmpDefault = '', self.defaultKibanaDashboardDir
|
||||
while (len(tmpVal) == 0):
|
||||
tmpVal = AskForString("Enter directory containing Kibana dashboards [{}]".format(tmpDefault), default=tmpDefault, acceptDefault=self.acceptDefaults)
|
||||
if (len(tmpVal) == 0): tmpVal = tmpDefault
|
||||
self.keystoreItems[BEAT_KIBANA_DASHBOARDS_PATH] = tmpVal
|
||||
|
||||
# username
|
||||
tmpVal, tmpDefault = '', ''
|
||||
while (len(tmpVal) == 0):
|
||||
tmpVal = AskForString("Enter HTTP/HTTPS server username", default=tmpDefault, acceptDefault=self.acceptDefaults)
|
||||
self.keystoreItems[BEAT_HTTP_USERNAME] = tmpVal
|
||||
|
||||
# password
|
||||
tmpVal, tmpValConfirm = '', 'xxxx'
|
||||
while (len(tmpVal) == 0) and (tmpVal != tmpValConfirm):
|
||||
tmpVal = AskForPassword("Enter password for {}: ".format(self.keystoreItems[BEAT_HTTP_USERNAME]))
|
||||
tmpValConfirm = AskForPassword("Enter password for {} (again): ".format(self.keystoreItems[BEAT_HTTP_USERNAME]))
|
||||
if (tmpVal != tmpValConfirm):
|
||||
eprint('Passwords do not match')
|
||||
tmpVal, tmpValConfirm = '', 'xxxx'
|
||||
self.keystoreItems[BEAT_HTTP_PASSWORD] = tmpVal
|
||||
|
||||
# write values to keystore
|
||||
for key, value in self.keystoreItems.items():
|
||||
err, out = self.run_beat_command(['keystore', 'add', key, '--stdin', '--force'], stdin=value)
|
||||
if (err != 0):
|
||||
raise Exception("Failed to add {} to {} keystore: {}".format(key, self.beatName, out))
|
||||
|
||||
# list keystore
|
||||
err, out = self.run_beat_command(['keystore', 'list'])
|
||||
if (err == 0):
|
||||
eprint('Generated keystore for {}'.format(self.beatName))
|
||||
eprint('\n'.join(out))
|
||||
else:
|
||||
raise Exception("Failed to enumerate keystore: {}".format(out))
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def beat_run(self):
|
||||
if self.debug:
|
||||
eprint("{}: {}".format(self.__class__.__name__, inspect.currentframe().f_code.co_name))
|
||||
|
||||
process = Popen(self.build_beat_command(['run', '-e']), stdout=PIPE)
|
||||
while True:
|
||||
output = process.stdout.readline()
|
||||
if (len(output) == 0) and (process.poll() is not None):
|
||||
break
|
||||
if output:
|
||||
print(output.decode().strip())
|
||||
else:
|
||||
time.sleep(0.5)
|
||||
process.poll()
|
||||
|
||||
###################################################################################################
|
||||
class LinuxBeatbox(Beatbox):
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def __init__(self, debug=False, ymlFileSpec=None, beatName=None):
|
||||
if PY3:
|
||||
super().__init__(debug=debug, ymlFileSpec=ymlFileSpec, beatName=beatName)
|
||||
else:
|
||||
super(LinuxBeatbox, self).__init__(debug=debug, ymlFileSpec=ymlFileSpec, beatName=beatName)
|
||||
|
||||
if not Which(self.beatExe, debug=self.debug):
|
||||
self.beatExe = self.beatExe.lower() if (self.beatExe is not None) else self.beatName.lower()
|
||||
|
||||
self.beatInstallDir = "/usr/share/{}".format(self.beatName)
|
||||
self.defaultKibanaDashboardDir = os.path.join(self.beatInstallDir, 'kibana')
|
||||
self.keystorePath = os.path.join(os.path.join(os.path.dirname(os.path.abspath(self.ymlFileSpec)), 'data'), "{}.keystore".format(self.beatName))
|
||||
|
||||
self.distro = None
|
||||
self.codename = None
|
||||
self.release = None
|
||||
|
||||
# determine the distro (e.g., ubuntu) and code name (e.g., bionic) if applicable
|
||||
|
||||
# check /etc/os-release values first
|
||||
if os.path.isfile('/etc/os-release'):
|
||||
osInfo = dict()
|
||||
|
||||
with open("/etc/os-release", 'r') as f:
|
||||
for line in f:
|
||||
try:
|
||||
k, v = line.rstrip().split("=")
|
||||
osInfo[k] = v.strip('"')
|
||||
except:
|
||||
pass
|
||||
|
||||
if ('NAME' in osInfo) and (len(osInfo['NAME']) > 0):
|
||||
distro = osInfo['NAME'].lower().split()[0]
|
||||
|
||||
if ('VERSION_CODENAME' in osInfo) and (len(osInfo['VERSION_CODENAME']) > 0):
|
||||
codename = osInfo['VERSION_CODENAME'].lower().split()[0]
|
||||
|
||||
if ('VERSION_ID' in osInfo) and (len(osInfo['VERSION_ID']) > 0):
|
||||
release = osInfo['VERSION_ID'].lower().split()[0]
|
||||
|
||||
# try lsb_release next
|
||||
if (self.distro is None):
|
||||
err, out = self.run_process(['lsb_release', '-is'], stderr=False)
|
||||
if (err == 0) and (len(out) > 0):
|
||||
self.distro = out[0].lower()
|
||||
|
||||
if (self.codename is None):
|
||||
err, out = self.run_process(['lsb_release', '-cs'], stderr=False)
|
||||
if (err == 0) and (len(out) > 0):
|
||||
self.codename = out[0].lower()
|
||||
|
||||
if (self.release is None):
|
||||
err, out = self.run_process(['lsb_release', '-rs'], stderr=False)
|
||||
if (err == 0) and (len(out) > 0):
|
||||
self.release = out[0].lower()
|
||||
|
||||
# try release-specific files
|
||||
if (self.distro is None):
|
||||
if os.path.isfile('/etc/centos-release'):
|
||||
distroFile = '/etc/centos-release'
|
||||
if os.path.isfile('/etc/redhat-release'):
|
||||
distroFile = '/etc/redhat-release'
|
||||
elif os.path.isfile('/etc/issue'):
|
||||
distroFile = '/etc/issue'
|
||||
else:
|
||||
distroFile = None
|
||||
if (distroFile is not None):
|
||||
with open(distroFile, 'r') as f:
|
||||
distroVals = f.read().lower().split()
|
||||
distroNums = [x for x in distroVals if x[0].isdigit()]
|
||||
self.distro = distroVals[0]
|
||||
if (self.release is None) and (len(distroNums) > 0):
|
||||
self.release = distroNums[0]
|
||||
|
||||
if (self.distro is None):
|
||||
self.distro = "linux"
|
||||
|
||||
if self.debug:
|
||||
eprint("distro: {}{}{}".format(self.distro,
|
||||
" {}".format(self.codename) if self.codename else "",
|
||||
" {}".format(self.release) if self.release else ""))
|
||||
|
||||
if not self.codename: self.codename = self.distro
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_beat_yml(self):
|
||||
if PY3:
|
||||
super().configure_beat_yml()
|
||||
else:
|
||||
super(LinuxBeatbox, self).configure_beat_yml()
|
||||
|
||||
localModulePath = os.path.join(os.path.abspath(self.ymlFilePath), 'module')
|
||||
installedModulePath = os.path.join(self.beatInstallDir, 'module')
|
||||
if ((not os.path.exists(localModulePath)) and
|
||||
(os.path.isdir(installedModulePath)) and
|
||||
YesOrNo("Create symlink to module path {} as {}?".format(installedModulePath, localModulePath), default=True, acceptDefault=self.acceptDefaults)):
|
||||
os.symlink(installedModulePath, localModulePath)
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_keystore(self):
|
||||
if PY3:
|
||||
super().configure_keystore()
|
||||
else:
|
||||
super(LinuxBeatbox, self).configure_keystore()
|
||||
|
||||
pass
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def beat_run(self):
|
||||
if PY3:
|
||||
super().beat_run()
|
||||
else:
|
||||
super(LinuxBeatbox, self).beat_run()
|
||||
|
||||
pass
|
||||
|
||||
###################################################################################################
|
||||
class WindowsBeatbox(Beatbox):
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def __init__(self, debug=False, ymlFileSpec=None, beatName=None):
|
||||
if PY3:
|
||||
super().__init__(debug=debug, ymlFileSpec=ymlFileSpec, beatName=beatName)
|
||||
else:
|
||||
super(WindowsBeatbox, self).__init__(debug=debug, ymlFileSpec=ymlFileSpec, beatName=beatName)
|
||||
|
||||
if not Which(self.beatExe, debug=self.debug):
|
||||
self.beatExe = self.beatExe + '.exe' if (self.beatExe is not None) else self.beatName + '.exe'
|
||||
|
||||
self.beatInstallDir = os.path.abspath(self.ymlFilePath)
|
||||
self.defaultKibanaDashboardDir = os.path.join(self.beatInstallDir, 'kibana')
|
||||
self.keystorePath = os.path.join(os.path.dirname(os.path.abspath(self.ymlFileSpec)), "{}.keystore".format(self.beatName))
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_beat_yml(self):
|
||||
if PY3:
|
||||
super().configure_beat_yml()
|
||||
else:
|
||||
super(WindowsBeatbox, self).configure_beat_yml()
|
||||
|
||||
pass
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_keystore(self):
|
||||
if PY3:
|
||||
super().configure_keystore()
|
||||
else:
|
||||
super(WindowsBeatbox, self).configure_keystore()
|
||||
|
||||
pass
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def beat_run(self):
|
||||
if PY3:
|
||||
super().beat_run()
|
||||
else:
|
||||
super(WindowsBeatbox, self).beat_run()
|
||||
|
||||
pass
|
||||
|
||||
###################################################################################################
|
||||
class MacBeatbox(Beatbox):
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def __init__(self, debug=False, ymlFileSpec=None, beatName=None):
|
||||
if PY3:
|
||||
super().__init__(debug=debug, ymlFileSpec=ymlFileSpec, beatName=beatName)
|
||||
else:
|
||||
super(MacBeatbox, self).__init__(debug=debug, ymlFileSpec=ymlFileSpec, beatName=beatName)
|
||||
|
||||
if not Which(self.beatExe, debug=self.debug):
|
||||
self.beatExe = self.beatExe.lower() if (self.beatExe is not None) else self.beatName.lower()
|
||||
|
||||
self.beatInstallDir = "/Library/Application Support/elastic/{}".format(self.beatName)
|
||||
self.defaultKibanaDashboardDir = os.path.join(self.beatInstallDir, 'kibana')
|
||||
self.keystorePath = os.path.join(os.path.join(os.path.dirname(os.path.abspath(self.ymlFileSpec)), 'data'), "{}.keystore".format(self.beatName))
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_beat_yml(self):
|
||||
if PY3:
|
||||
super().configure_beat_yml()
|
||||
else:
|
||||
super(MacBeatbox, self).configure_beat_yml()
|
||||
|
||||
localModulePath = os.path.join(os.path.abspath(self.ymlFilePath), 'module')
|
||||
installedModulePath = os.path.join(self.beatInstallDir, 'module')
|
||||
if ((not os.path.exists(localModulePath)) and
|
||||
(os.path.isdir(installedModulePath)) and
|
||||
YesOrNo("Create symlink to module path {} as {}?".format(installedModulePath, localModulePath), default=True, acceptDefault=self.acceptDefaults)):
|
||||
os.symlink(installedModulePath, localModulePath)
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def configure_keystore(self):
|
||||
if PY3:
|
||||
super().configure_keystore()
|
||||
else:
|
||||
super(MacBeatbox, self).configure_keystore()
|
||||
|
||||
pass
|
||||
|
||||
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
def beat_run(self):
|
||||
if PY3:
|
||||
super().beat_run()
|
||||
else:
|
||||
super(MacBeatbox, self).beat_run()
|
||||
|
||||
pass
|
||||
65
Vagrant/resources/malcolm/scripts/beats/beat_config.py
Executable file
65
Vagrant/resources/malcolm/scripts/beats/beat_config.py
Executable file
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
from beat_common import *
|
||||
|
||||
###################################################################################################
|
||||
ScriptName = os.path.basename(__file__)
|
||||
|
||||
###################################################################################################
|
||||
# main
|
||||
def main():
|
||||
|
||||
# extract arguments from the command line
|
||||
# print (sys.argv[1:]);
|
||||
parser = argparse.ArgumentParser(description='Beat configure script', add_help=False, usage='{} <arguments>'.format(ScriptName))
|
||||
parser.add_argument('-v', '--verbose', dest='debug', type=str2bool, nargs='?', const=True, default=False, help="Verbose output")
|
||||
parser.add_argument('-b', '--beat', required=True, dest='beatName', metavar='<STR>', type=str, default=None, help='Beat name')
|
||||
parser.add_argument('-c', '--config-file', required=False, dest='configFile', metavar='<STR>', type=str, default=None, help='Beat YML file to configure')
|
||||
parser.add_argument('-d', '--defaults', dest='acceptDefault', type=str2bool, nargs='?', const=True, default=False, help="Accept defaults to prompts without user interaction")
|
||||
|
||||
try:
|
||||
parser.error = parser.exit
|
||||
args = parser.parse_args()
|
||||
except SystemExit:
|
||||
parser.print_help()
|
||||
exit(2)
|
||||
|
||||
if args.debug:
|
||||
eprint(os.path.join(ScriptPath, ScriptName))
|
||||
eprint("Arguments: {}".format(sys.argv[1:]))
|
||||
eprint("Arguments: {}".format(args))
|
||||
else:
|
||||
sys.tracebacklimit = 0
|
||||
|
||||
args.beatName = args.beatName.lower()
|
||||
if not args.beatName.endswith('beat'):
|
||||
args.beatName = args.beatName + 'beat'
|
||||
|
||||
if (args.configFile is None):
|
||||
args.configFile = args.beatName + '.yml'
|
||||
|
||||
installerPlatform = platform.system()
|
||||
if installerPlatform == PLATFORM_LINUX:
|
||||
Beatbox = LinuxBeatbox(debug=args.debug, ymlFileSpec=args.configFile, beatName=args.beatName)
|
||||
elif installerPlatform == PLATFORM_MAC:
|
||||
Beatbox = MacBeatbox(debug=args.debug, ymlFileSpec=args.configFile, beatName=args.beatName)
|
||||
elif installerPlatform == PLATFORM_WINDOWS:
|
||||
Beatbox = WindowsBeatbox(debug=args.debug, ymlFileSpec=args.configFile, beatName=args.beatName)
|
||||
|
||||
success = False
|
||||
if hasattr(Beatbox, 'configure_beat_yml'): success = Beatbox.configure_beat_yml()
|
||||
if hasattr(Beatbox, 'configure_keystore'): success = Beatbox.configure_keystore()
|
||||
|
||||
return success
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
63
Vagrant/resources/malcolm/scripts/beats/beat_run.py
Executable file
63
Vagrant/resources/malcolm/scripts/beats/beat_run.py
Executable file
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
from beat_common import *
|
||||
|
||||
###################################################################################################
|
||||
ScriptName = os.path.basename(__file__)
|
||||
|
||||
###################################################################################################
|
||||
# main
|
||||
def main():
|
||||
|
||||
# extract arguments from the command line
|
||||
# print (sys.argv[1:]);
|
||||
parser = argparse.ArgumentParser(description='Beat local execution script', add_help=False, usage='{} <arguments>'.format(ScriptName))
|
||||
parser.add_argument('-v', '--verbose', dest='debug', type=str2bool, nargs='?', const=True, default=False, help="Verbose output")
|
||||
parser.add_argument('-b', '--beat', required=True, dest='beatName', metavar='<STR>', type=str, default=None, help='Beat name')
|
||||
parser.add_argument('-c', '--config-file', required=False, dest='configFile', metavar='<STR>', type=str, default=None, help='Beat YML config file')
|
||||
|
||||
try:
|
||||
parser.error = parser.exit
|
||||
args = parser.parse_args()
|
||||
except SystemExit:
|
||||
parser.print_help()
|
||||
exit(2)
|
||||
|
||||
if args.debug:
|
||||
eprint(os.path.join(ScriptPath, ScriptName))
|
||||
eprint("Arguments: {}".format(sys.argv[1:]))
|
||||
eprint("Arguments: {}".format(args))
|
||||
else:
|
||||
sys.tracebacklimit = 0
|
||||
|
||||
args.beatName = args.beatName.lower()
|
||||
if not args.beatName.endswith('beat'):
|
||||
args.beatName = args.beatName + 'beat'
|
||||
|
||||
if (args.configFile is None):
|
||||
args.configFile = args.beatName + '.yml'
|
||||
|
||||
installerPlatform = platform.system()
|
||||
if installerPlatform == PLATFORM_LINUX:
|
||||
Beatbox = LinuxBeatbox(debug=args.debug, ymlFileSpec=args.configFile, beatName=args.beatName)
|
||||
elif installerPlatform == PLATFORM_MAC:
|
||||
Beatbox = MacBeatbox(debug=args.debug, ymlFileSpec=args.configFile, beatName=args.beatName)
|
||||
elif installerPlatform == PLATFORM_WINDOWS:
|
||||
Beatbox = WindowsBeatbox(debug=args.debug, ymlFileSpec=args.configFile, beatName=args.beatName)
|
||||
|
||||
success = False
|
||||
if hasattr(Beatbox, 'beat_run'): success = Beatbox.beat_run()
|
||||
|
||||
return success
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
89
Vagrant/resources/malcolm/scripts/beats/linux_vm_example/Vagrantfile
vendored
Normal file
89
Vagrant/resources/malcolm/scripts/beats/linux_vm_example/Vagrantfile
vendored
Normal file
@@ -0,0 +1,89 @@
|
||||
# -*- mode: ruby -*-
|
||||
# vi: set ft=ruby :
|
||||
|
||||
unless Vagrant.has_plugin?("vagrant-reload")
|
||||
raise 'vagrant-reload plugin is not installed!'
|
||||
end
|
||||
|
||||
# hack: https://github.com/hashicorp/vagrant/issues/8878#issuecomment-345112810
|
||||
class VagrantPlugins::ProviderVirtualBox::Action::Network
|
||||
def dhcp_server_matches_config?(dhcp_server, config)
|
||||
true
|
||||
end
|
||||
end
|
||||
|
||||
Vagrant.configure("2") do |config|
|
||||
|
||||
config.vm.box = "bento/ubuntu-20.04"
|
||||
|
||||
config.vm.network "private_network", type: "dhcp"
|
||||
|
||||
config.vm.synced_folder ".", "/vagrant", disabled: true
|
||||
|
||||
if Vagrant.has_plugin?("vagrant-vbguest")
|
||||
config.vbguest.auto_update = false
|
||||
end
|
||||
|
||||
config.vm.provider "virtualbox" do |vb|
|
||||
vb.customize ["modifyvm", :id, "--nictype1", "virtio" ]
|
||||
vb.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
|
||||
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
|
||||
vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
|
||||
vb.customize ["modifyvm", :id, "--memory", 2048]
|
||||
vb.customize ["modifyvm", :id, "--cpus", 2]
|
||||
vb.customize ["modifyvm", :id, "--vram", 32]
|
||||
vb.customize ["modifyvm", :id, "--ioapic", "on"]
|
||||
vb.customize ["modifyvm", :id, "--nestedpaging", "on"]
|
||||
vb.customize ["modifyvm", :id, "--pae", "on"]
|
||||
vb.customize ["modifyvm", :id, "--hwvirtex", "on"]
|
||||
vb.customize ["modifyvm", :id, "--nested-hw-virt", "on"]
|
||||
end
|
||||
|
||||
config.vm.provision "shell", inline: <<-STEP1
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
export BEAT_VERSION=7.6.2
|
||||
apt-get update
|
||||
apt-get install -y auditd gnupg2 curl ca-certificates libcap2-bin libpcap0.8 python3-minimal python-is-python3
|
||||
curl -sSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
|
||||
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" >> /etc/apt/sources.list
|
||||
apt-get update
|
||||
for BEAT in auditbeat filebeat packetbeat metricbeat; do
|
||||
apt-get install -y $BEAT-oss=$BEAT_VERSION
|
||||
done;
|
||||
STEP1
|
||||
|
||||
config.vm.provision "file", source: "./audit.rules", destination: "/tmp/audit.rules"
|
||||
|
||||
config.vm.provision "file", source: "../beat_run.py", destination: "/tmp/beat_run.py"
|
||||
config.vm.provision "file", source: "../beat_config.py", destination: "/tmp/beat_config.py"
|
||||
config.vm.provision "file", source: "../beat_common.py", destination: "/tmp/beat_common.py"
|
||||
["auditbeat","filebeat","packetbeat","metricbeat"].to_enum.with_index(1).each do |beat, i|
|
||||
config.vm.provision "file", source: "./#{beat}.yml", destination: "/tmp/#{beat}.yml"
|
||||
end
|
||||
|
||||
config.vm.provision "shell", inline: <<-STEP2
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
mv /tmp/beat*.py /usr/local/bin/
|
||||
chown root:root /usr/local/bin/beat*.py
|
||||
chmod 755 /usr/local/bin/beat_config.py /usr/local/bin/beat_run.py
|
||||
chmod 644 /usr/local/bin/beat_common.py
|
||||
|
||||
filebeat modules enable system
|
||||
|
||||
mv /tmp/audit.rules /etc/audit/rules.d/audit.rules
|
||||
find /etc/audit -type d -exec chmod 750 "{}" \\;
|
||||
find /etc/audit -type f -exec chmod 640 "{}" \\;
|
||||
|
||||
for BEAT in auditbeat filebeat packetbeat metricbeat; do
|
||||
mkdir -p /opt/$BEAT
|
||||
mv /tmp/$BEAT.yml /opt/$BEAT/
|
||||
chown -R root:root /opt/$BEAT
|
||||
chmod 700 /opt/$BEAT
|
||||
chmod 600 /opt/$BEAT/*
|
||||
done;
|
||||
STEP2
|
||||
|
||||
config.vm.provision :reload
|
||||
|
||||
end
|
||||
@@ -0,0 +1,146 @@
|
||||
## First rule - delete all
|
||||
-D
|
||||
|
||||
## Increase the buffers to survive stress events.
|
||||
## Make this bigger for busy systems
|
||||
-b 8192
|
||||
|
||||
## This determine how long to wait in burst of events
|
||||
--backlog_wait_time 0
|
||||
|
||||
## Set failure mode to syslog
|
||||
-f 1
|
||||
|
||||
# exclusions
|
||||
|
||||
-a always,exclude -F msgtype=AVC
|
||||
-a always,exclude -F msgtype=CRYPTO_KEY_USER
|
||||
-a always,exclude -F msgtype=CWD
|
||||
-a always,exclude -F msgtype=EOE
|
||||
|
||||
# commands
|
||||
|
||||
-a always,exit -F path=/bin/fusermount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/bin/mount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/bin/pmount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/bin/pumount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/bin/su -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/bin/umount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/etc/audisp/audisp-remote.conf -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/default/grub -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/fstab -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/hosts.deny -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/login.defs -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/profile -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/etc/sysctl.conf -F perm=wa -k config_file_change
|
||||
-a always,exit -F path=/sbin/apparmor_parser -F perm=x -F auid>=1000 -F auid!=4294967295 -k MAC-policy
|
||||
-a always,exit -F path=/sbin/pam_tally -F perm=wxa -F auid>=1000 -F auid!=4294967295 -k privileged-pam
|
||||
-a always,exit -F path=/sbin/pam_tally2 -F perm=wxa -F auid>=1000 -F auid!=4294967295 -k privileged-pam
|
||||
-a always,exit -F path=/sbin/unix_chkpwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-passwd
|
||||
-a always,exit -F path=/sbin/unix_update -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-unix-update
|
||||
-a always,exit -F path=/usr/bin/bsd-write -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/chacl -F perm=x -F auid>=1000 -F auid!=4294967295 -k perm_chng
|
||||
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-passwd
|
||||
-a always,exit -F path=/usr/bin/chcon -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-cron
|
||||
-a always,exit -F path=/usr/bin/dotlock.mailutils -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/expiry -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/fusermount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-passwd
|
||||
-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/usr/bin/ntfs-3g -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-passwd
|
||||
-a always,exit -F path=/usr/bin/pkexec -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/pmount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/pumount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/setfacl -F perm=x -F auid>=1000 -F auid!=4294967295 -k perm_chng
|
||||
-a always,exit -F path=/usr/bin/ssh-agent -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-ssh
|
||||
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/usr/bin/sudoedit -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-priv_change
|
||||
-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/bin/wall -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/lib/dbus-1.0/dbus-daemon-launch-helper -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/lib/eject/dmcrypt-get-device -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/lib/openssh/ssh-keysign -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-ssh
|
||||
-a always,exit -F path=/usr/lib/policykit-1/polkit-agent-helper-1 -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/lib/x86_64-linux-gnu/utempter/utempter -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/lib/xorg/Xorg.wrap -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/addgroup -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/adduser -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/exim4 -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/groupadd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/mount.cifs -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/netfilter-persistent -F perm=x -F auid>=1000 -F auid!=4294967295 -k nft_persistent_use
|
||||
-a always,exit -F path=/usr/sbin/nft -F perm=x -F auid>=1000 -F auid!=4294967295 -k nft_cmd_use
|
||||
-a always,exit -F path=/usr/sbin/pam_timestamp_check -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-pam
|
||||
-a always,exit -F path=/usr/sbin/postdrop -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-postfix
|
||||
-a always,exit -F path=/usr/sbin/postqueue -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-postfix
|
||||
-a always,exit -F path=/usr/sbin/semanage -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/setsebool -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/unix_chkpwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/useradd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/userhelper -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
-a always,exit -F path=/usr/sbin/usermod -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged-usermod
|
||||
-a always,exit -F path=/usr/sbin/visudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged
|
||||
|
||||
# privileged files
|
||||
|
||||
-w /bin/kmod -p x -k modules
|
||||
-w /etc/apparmor.d/ -p wa -k MAC-policy
|
||||
-w /etc/apparmor/ -p wa -k MAC-policy
|
||||
-w /etc/group -p wa -k identity
|
||||
-w /etc/gshadow -p wa -k identity
|
||||
-w /etc/hosts -p wa -k system-locale
|
||||
-w /etc/issue -p wa -k system-locale
|
||||
-w /etc/issue.net -p wa -k system-locale
|
||||
-w /etc/localtime -p wa -k time-change
|
||||
-w /etc/network -p wa -k system-locale
|
||||
-w /etc/nftables.conf -p wa -k nft_config_file_change
|
||||
-w /etc/opasswd -p wa -k usergroup_modification
|
||||
-w /etc/passwd -p wa -k identity
|
||||
-w /etc/security/opasswd -p wa -k identity
|
||||
-w /etc/shadow -p wa -k identity
|
||||
-w /etc/sudoers -p wa -k sudoers
|
||||
-w /etc/sudoers.d/ -p wa -k sudoers
|
||||
-w /sbin/insmod -p x -k modules
|
||||
-w /sbin/modprobe -p x -k modules
|
||||
-w /sbin/rmmod -p x -k modules
|
||||
-w /var/log/btmp -p wa -k session
|
||||
-w /var/log/faillog -p wa -k logins
|
||||
-w /var/log/lastlog -p wa -k logins
|
||||
-w /var/log/sudo.log -p wa -k sudoaction
|
||||
-w /var/log/tallylog -p wa -k logins
|
||||
-w /var/log/wtmp -p wa -k session
|
||||
-w /var/run/faillock -p wa -k logins
|
||||
-w /var/run/utmp -p wa -k session
|
||||
|
||||
# syscalls
|
||||
|
||||
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change
|
||||
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=1000 -F auid!=4294967295 -k perm_mod
|
||||
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F auid>=1000 -F auid!=4294967295 -k perm_mod
|
||||
-a always,exit -F arch=b64 -S clock_settime -k time-change
|
||||
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=4294967295 -k access
|
||||
-a always,exit -F arch=b64 -S open_by_handle_at -F exit=-EACCES -F auid>=1000 -F auid!=4294967295 -k access
|
||||
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=4294967295 -k access
|
||||
-a always,exit -F arch=b64 -S open_by_handle_at -F exit=-EPERM -F auid>=1000 -F auid!=4294967295 -k access
|
||||
-a always,exit -F arch=b64 -S execve -C gid!=egid -F key=execpriv
|
||||
-a always,exit -F arch=b64 -S execve -C uid!=euid -F key=execpriv
|
||||
-a always,exit -F arch=b64 -S init_module -S delete_module -S create_module -S finit_module -k modules
|
||||
-a always,exit -F arch=b64 -S mount -F auid>=1000 -F auid!=4294967295 -k mounts
|
||||
-a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F auid!=4294967295 -k perm_mod
|
||||
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -S rmdir -F auid>=1000 -F auid!=4294967295 -k delete
|
||||
-a always,exit -F dir=/etc/audit/rules.d/ -F perm=wa -k config_file_change
|
||||
-a always,exit -F dir=/etc/pam.d/ -F perm=wa -k config_file_change
|
||||
-a always,exit -F dir=/etc/profile.d/ -F perm=wa -k config_file_change
|
||||
-a always,exit -F dir=/etc/security/ -F perm=wa -k config_file_change
|
||||
-a exit,always -F arch=b64 -S sethostname -S setdomainname -k system-locale
|
||||
|
||||
# Make the configuration immutable -- reboot is required to change audit rules
|
||||
-e 2
|
||||
@@ -0,0 +1,154 @@
|
||||
# See https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-reference-yml.html
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
auditbeat.modules:
|
||||
|
||||
#------------------------------- auditd Module -------------------------------
|
||||
|
||||
- module: auditd
|
||||
socket_type: multicast
|
||||
resolve_ids: true
|
||||
failure_mode: log
|
||||
backlog_limit: 16384
|
||||
rate_limit: 0
|
||||
include_raw_message: false
|
||||
include_warnings: false
|
||||
backpressure_strategy: auto
|
||||
# audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
|
||||
# no rules specified, auditd will run and manage rules
|
||||
# see https://www.elastic.co/guide/en/beats/auditbeat/master/auditbeat-module-auditd.html
|
||||
|
||||
# don't forward some things that are always going to be happening to cut down on noise
|
||||
# and some other approved common stuff that would clutter the logs
|
||||
processors:
|
||||
- drop_event:
|
||||
when:
|
||||
and:
|
||||
- equals:
|
||||
auditd.message_type: 'syscall'
|
||||
- equals:
|
||||
auditd.summary.object.type: 'file'
|
||||
- or:
|
||||
- and:
|
||||
- or:
|
||||
- equals:
|
||||
auditd.data.syscall: 'open'
|
||||
- equals:
|
||||
auditd.data.syscall: 'openat'
|
||||
- regexp:
|
||||
auditd.summary.object.primary: '^/(proc/|etc/localtime|usr/lib/x86_64-linux-gnu/gconv/gconv-modules\.cache)'
|
||||
- or:
|
||||
- equals:
|
||||
auditd.summary.how: '/usr/share/auditbeat/bin/auditbeat'
|
||||
- and:
|
||||
- or:
|
||||
- equals:
|
||||
auditd.data.syscall: 'open'
|
||||
- equals:
|
||||
auditd.data.syscall: 'openat'
|
||||
- not:
|
||||
has_fields: ['auditd.summary.object.primary']
|
||||
- and:
|
||||
- equals:
|
||||
auditd.data.syscall: 'open'
|
||||
- regexp:
|
||||
auditd.summary.object.primary: '^/.+/__pycache__/$'
|
||||
- equals:
|
||||
auditd.summary.how: 'python3.8'
|
||||
|
||||
|
||||
- module: file_integrity
|
||||
paths:
|
||||
- /bin
|
||||
- /etc
|
||||
- /sbin
|
||||
- /usr/bin
|
||||
- /usr/local/bin
|
||||
- /usr/sbin
|
||||
recursive: true
|
||||
|
||||
# TODO: system module is apparently only available in the non-OSS basic license :-(
|
||||
|
||||
# - module: system
|
||||
# datasets:
|
||||
# - host # General host information, e.g. uptime, IPs
|
||||
# - user # User information
|
||||
# - login # Logins/logouts
|
||||
# - package # dpkg/rpm package manager logs
|
||||
# period: 1m
|
||||
# state.period: 12h
|
||||
# user.detect_password_changes: true
|
||||
|
||||
# - module: system
|
||||
# datasets:
|
||||
# - process # Started and stopped processes
|
||||
# - socket # Opened and closed sockets
|
||||
# period: 1s
|
||||
|
||||
# # drop noise
|
||||
# processors:
|
||||
# - drop_event:
|
||||
# when:
|
||||
# or:
|
||||
# - and:
|
||||
# - equals:
|
||||
# event.module: 'system'
|
||||
# - equals:
|
||||
# event.dataset: 'socket'
|
||||
# - equals:
|
||||
# destination.ip: '127.0.0.1'
|
||||
# - equals:
|
||||
# source.ip: '127.0.0.1'
|
||||
# - and:
|
||||
# - equals:
|
||||
# event.module: 'system'
|
||||
# - equals:
|
||||
# event.dataset: 'socket'
|
||||
# - equals:
|
||||
# destination.ip: "${BEAT_ES_HOST}"
|
||||
# - and:
|
||||
# - equals:
|
||||
# event.module: 'system'
|
||||
# - equals:
|
||||
# event.dataset: 'socket'
|
||||
# - equals:
|
||||
# destination.ip: "${BEAT_KIBANA_HOST}"
|
||||
# - and:
|
||||
# - equals:
|
||||
# event.module: 'system'
|
||||
# - equals:
|
||||
# event.dataset: 'process'
|
||||
# - or:
|
||||
# - equals:
|
||||
# process.executable: '/bin/sleep'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/sort'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/tail'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/clear'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/head'
|
||||
# - equals:
|
||||
# process.executable: '/bin/date'
|
||||
# - equals:
|
||||
# process.executable: '/bin/ls'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/stat'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/cut'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/xargs'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/tr'
|
||||
# - equals:
|
||||
# process.executable: '/bin/grep'
|
||||
# - equals:
|
||||
# process.executable: '/bin/sed'
|
||||
# - equals:
|
||||
# process.executable: '/bin/df'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/du'
|
||||
# - equals:
|
||||
# process.executable: '/usr/bin/gawk'
|
||||
@@ -0,0 +1,14 @@
|
||||
# See https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html
|
||||
# https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
filebeat.modules:
|
||||
|
||||
#------------------------------- System Module -------------------------------
|
||||
|
||||
- module: system
|
||||
syslog:
|
||||
enabled: true
|
||||
auth:
|
||||
enabled: true
|
||||
@@ -0,0 +1,44 @@
|
||||
# See https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-reference-yml.html
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
metricbeat.config.modules:
|
||||
path: ${path.config}/conf.d/*.yml
|
||||
reload.period: 10s
|
||||
reload.enabled: false
|
||||
|
||||
metricbeat.max_start_delay: 10s
|
||||
|
||||
metricbeat.modules:
|
||||
|
||||
#------------------------------- System Module -------------------------------
|
||||
|
||||
- module: system
|
||||
period: 30s
|
||||
metricsets:
|
||||
- cpu # CPU usage
|
||||
- load # CPU load averages
|
||||
- memory # Memory usage
|
||||
- network # Network IO
|
||||
- process # Per process metrics
|
||||
- process_summary # Process summary
|
||||
- uptime # System Uptime
|
||||
- diskio # Disk IO
|
||||
enabled: true
|
||||
processes: ['.*']
|
||||
process.include_top_n:
|
||||
enabled: true
|
||||
by_cpu: 10
|
||||
by_memory: 10
|
||||
|
||||
cpu.metrics: ["percentages"]
|
||||
core.metrics: ["percentages"]
|
||||
|
||||
- module: system
|
||||
period: 1m
|
||||
metricsets:
|
||||
- filesystem # File system usage for each mountpoint
|
||||
- fsstat # File system summary metrics
|
||||
processors:
|
||||
- drop_event.when.regexp:
|
||||
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|boot)($|/)'
|
||||
@@ -0,0 +1,87 @@
|
||||
# See https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-reference-yml.html
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
#------------------------------- network device ------------------------------
|
||||
|
||||
packetbeat.interfaces.device: any
|
||||
packetbeat.interfaces.type: pcap
|
||||
packetbeat.interfaces.snaplen: 65535
|
||||
|
||||
#------------------------------- flows ---------------------------------------
|
||||
|
||||
packetbeat.flows:
|
||||
enabled: true
|
||||
timeout: 30s
|
||||
period: 10s
|
||||
|
||||
#------------------------------- transaction protocols -----------------------
|
||||
|
||||
packetbeat.protocols:
|
||||
- type: icmp
|
||||
enabled: true
|
||||
|
||||
- type: amqp
|
||||
enabled: true
|
||||
ports: [5672]
|
||||
|
||||
- type: cassandra
|
||||
enabled: true
|
||||
ports: [9042]
|
||||
|
||||
- type: dhcpv4
|
||||
enabled: true
|
||||
ports: [67, 68]
|
||||
|
||||
- type: dns
|
||||
enabled: true
|
||||
ports: [53]
|
||||
include_authorities: true
|
||||
include_additionals: true
|
||||
|
||||
- type: http
|
||||
enabled: true
|
||||
ports: [80, 8080, 8000, 5000, 8002]
|
||||
|
||||
- type: memcache
|
||||
enabled: true
|
||||
ports: [11211]
|
||||
|
||||
- type: mysql
|
||||
enabled: true
|
||||
ports: [3306,3307]
|
||||
|
||||
- type: pgsql
|
||||
enabled: true
|
||||
ports: [5432]
|
||||
|
||||
- type: redis
|
||||
enabled: true
|
||||
ports: [6379]
|
||||
|
||||
- type: thrift
|
||||
enabled: true
|
||||
ports: [9090]
|
||||
|
||||
- type: mongodb
|
||||
enabled: true
|
||||
ports: [27017]
|
||||
|
||||
- type: nfs
|
||||
enabled: true
|
||||
ports: [2049]
|
||||
|
||||
- type: tls
|
||||
enabled: true
|
||||
ports:
|
||||
- 443 # HTTPS
|
||||
- 993 # IMAPS
|
||||
- 995 # POP3S
|
||||
- 5223 # XMPP over SSL
|
||||
- 8883 # Secure MQTT
|
||||
- 9243 # Elasticsearch
|
||||
|
||||
#------------------------------- monitored processes -------------------------
|
||||
|
||||
packetbeat.procs.enabled: true
|
||||
packetbeat.ignore_outgoing: false
|
||||
@@ -0,0 +1,188 @@
|
||||
# configure a windows host to forward auditbeat and winlogbeat logs
|
||||
# to Malcolm (see https://github.com/idaholab/Malcolm/tree/master/scripts/beats)
|
||||
|
||||
$beatversion = "7.6.2"
|
||||
|
||||
################################################################################
|
||||
# Uninstall-Beat
|
||||
#
|
||||
# - Remove previous traces of this beat
|
||||
#
|
||||
function Uninstall-Beat {
|
||||
param( [string]$beat )
|
||||
|
||||
try {
|
||||
& "C:\\Program Files\\$beat\\uninstall-service-$beat.ps1"
|
||||
}
|
||||
catch {
|
||||
}
|
||||
remove-item "C:\\Program Files\\$beat" -Recurse -erroraction 'silentlycontinue';
|
||||
|
||||
}
|
||||
|
||||
################################################################################
|
||||
# Download-Beat
|
||||
#
|
||||
# - Download $beat-$beatversion-windows-x86_64.zip from artifacts.elastic.co
|
||||
# - Unzip to C:\Program Files\beat
|
||||
# - Download sample config for $beat from idaholab/Malcolm to C:\Program Files\beat
|
||||
#
|
||||
function Download-Beat {
|
||||
param( [string]$beat )
|
||||
|
||||
Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/$beat/$beat-oss-$beatversion-windows-x86_64.zip -OutFile $beat-$beatversion-windows-x86_64.zip -UseBasicParsing
|
||||
Expand-Archive -LiteralPath $beat-$beatversion-windows-x86_64.zip -DestinationPath 'C:\\Program Files'
|
||||
Remove-Item $beat-$beatversion-windows-x86_64.zip
|
||||
Rename-Item "C:\\Program Files\\$beat-$beatversion-windows-x86_64" "C:\\Program Files\\$beat"
|
||||
((Get-Content -path "C:\\Program Files\\$beat\\install-service-$beat.ps1" -Raw) -replace 'ProgramData','Program Files') | Set-Content -Path "C:\\Program Files\\$beat\\install-service-$beat.ps1"
|
||||
((Get-Content -path "C:\\Program Files\\$beat\\install-service-$beat.ps1" -Raw) -replace ' -path',' --path') | Set-Content -Path "C:\\Program Files\\$beat\\install-service-$beat.ps1"
|
||||
|
||||
Invoke-WebRequest -UseBasicParsing -OutFile "C:\\Program Files\\$beat\\$beat.yml" -Uri https://raw.githubusercontent.com/idaholab/Malcolm/master/scripts/beats/windows_vm_example/$beat.yml
|
||||
(Get-Content "C:\\Program Files\\$beat\\$beat.yml") | Set-Content "C:\\Program Files\\$beat\\$beat.yml"
|
||||
}
|
||||
|
||||
################################################################################
|
||||
# Connectivity boilerplate to add to the sample .yml files downloaded from
|
||||
# idaholab/Malcolm
|
||||
#
|
||||
$beat_boilerplate = @'
|
||||
|
||||
#================================ General ======================================
|
||||
fields_under_root: true
|
||||
|
||||
#================================ Outputs ======================================
|
||||
|
||||
#-------------------------- Elasticsearch output -------------------------------
|
||||
output.elasticsearch:
|
||||
enabled: true
|
||||
hosts: ["${BEAT_ES_HOST}"]
|
||||
protocol: "${BEAT_ES_PROTOCOL}"
|
||||
username: "${BEAT_HTTP_USERNAME}"
|
||||
password: "${BEAT_HTTP_PASSWORD}"
|
||||
ssl.verification_mode: "${BEAT_ES_SSL_VERIFY}"
|
||||
|
||||
setup.template.enabled: true
|
||||
setup.template.overwrite: false
|
||||
setup.template.settings:
|
||||
index.number_of_shards: 1
|
||||
index.number_of_replicas: 0
|
||||
|
||||
#============================== Dashboards =====================================
|
||||
setup.dashboards.enabled: "${BEAT_KIBANA_DASHBOARDS_ENABLED}"
|
||||
setup.dashboards.directory: "${BEAT_KIBANA_DASHBOARDS_PATH}"
|
||||
|
||||
#============================== Kibana =====================================
|
||||
setup.kibana:
|
||||
host: "${BEAT_KIBANA_HOST}"
|
||||
protocol: "${BEAT_KIBANA_PROTOCOL}"
|
||||
username: "${BEAT_HTTP_USERNAME}"
|
||||
password: "${BEAT_HTTP_PASSWORD}"
|
||||
ssl.verification_mode: "${BEAT_KIBANA_SSL_VERIFY}"
|
||||
|
||||
#================================ Logging ======================================
|
||||
logging.metrics.enabled: false
|
||||
'@
|
||||
|
||||
################################################################################
|
||||
# Run-Beat-Command
|
||||
#
|
||||
# - Run C:\Program Files\$beat\$beat.exe with correct defaults for config paths
|
||||
# - specify beat, command array and (optionally) stdin string
|
||||
#
|
||||
function Run-Beat-Command {
|
||||
param( [string]$beat, [array]$command, [string]$stdin)
|
||||
|
||||
$exe = "C:\\Program Files\\$beat\\$beat.exe"
|
||||
$exe_config = '--path.home', "C:\\Program Files\\$beat", '--path.config', "C:\\Program Files\\$beat", '--path.data', "C:\\Program Files\\$beat", '--path.logs', "C:\\Program Files\\$beat\\logs", '-c', "C:\\Program Files\\$beat\\$beat.yml", '-E', "keystore.path='C:\\Program Files\\$beat\\$beat.keystore'"
|
||||
|
||||
if (!$stdin) {
|
||||
& $exe $exe_config $command
|
||||
} else {
|
||||
$stdin.Trim() | & $exe $exe_config $command
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
################################################################################
|
||||
# Configure config .yml and keystore for beat in "C:\\Program Files\\$beat"
|
||||
#
|
||||
function Configure-Beat {
|
||||
param( [string]$beat )
|
||||
|
||||
cd "C:\\Program Files\\$beat"
|
||||
|
||||
Run-Beat-Command $beat @("keystore","create","--force") $null
|
||||
|
||||
Add-Content -Path "C:\\Program Files\\$beat\\$beat.yml" -Value $beat_boilerplate
|
||||
|
||||
do {
|
||||
$es_host = Read-Host "Specify the Elasticsearch IP:port (e.g., 192.168.0.123:9200)"
|
||||
$es_host = $es_host.Trim()
|
||||
} while (!$es_host)
|
||||
|
||||
do {
|
||||
$kb_host = Read-Host "Specify the Kibana IP:port (e.g., 192.168.0.123:5601)"
|
||||
$kb_host = $kb_host.Trim()
|
||||
} while (!$kb_host)
|
||||
|
||||
do {
|
||||
$es_user = Read-Host "Specify the Elasticsearch/Kibana username"
|
||||
$es_user = $es_user.Trim()
|
||||
} while (!$es_user)
|
||||
|
||||
do {
|
||||
$es_pass = Read-Host "Specify the Elasticsearch/Kibana password" -AsSecureString
|
||||
$es_pass_confirm = Read-Host "Specify the Elasticsearch/Kibana password (again)" -AsSecureString
|
||||
$pwd1_text = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($es_pass))
|
||||
$pwd2_text = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($es_pass_confirm))
|
||||
} while ($pwd1_text -ne $pwd2_text)
|
||||
$es_pass = ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($es_pass))).Trim()
|
||||
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_ES_PROTOCOL","--stdin","--force") "https"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_KIBANA_PROTOCOL","--stdin","--force") "https"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_ES_SSL_VERIFY","--stdin","--force") "none"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_KIBANA_SSL_VERIFY","--stdin","--force") "none"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_KIBANA_DASHBOARDS_ENABLED","--stdin","--force") "true"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_KIBANA_DASHBOARDS_PATH","--stdin","--force") "C:\\Program Files\\$beat\\kibana"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_ES_HOST","--stdin","--force") "$es_host"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_KIBANA_HOST","--stdin","--force") "$kb_host"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_HTTP_USERNAME","--stdin","--force") "$es_user"
|
||||
Run-Beat-Command $beat @("keystore","add","BEAT_HTTP_PASSWORD","--stdin","--force") "$es_pass"
|
||||
|
||||
Run-Beat-Command $beat @("keystore","list") $null
|
||||
|
||||
$confirmation = Read-Host "Install $beat as a system service (y/n)"
|
||||
if ($confirmation -eq 'y') {
|
||||
& "C:\\Program Files\\$beat\\install-service-$beat.ps1"
|
||||
}
|
||||
}
|
||||
|
||||
################################################################################
|
||||
# Main
|
||||
#
|
||||
function Main {
|
||||
param( [array]$beats)
|
||||
$tempdir = New-TemporaryFile
|
||||
remove-item $tempdir;
|
||||
new-item -type directory -path $tempdir;
|
||||
cd $tempdir;
|
||||
|
||||
foreach ($beat in $beats) {
|
||||
cd $tempdir;
|
||||
|
||||
Uninstall-Beat $beat
|
||||
Download-Beat $beat
|
||||
Configure-Beat $beat
|
||||
}
|
||||
|
||||
cd $Env:Temp;
|
||||
remove-item $tempdir -Recurse;
|
||||
}
|
||||
|
||||
################################################################################
|
||||
#
|
||||
if ($args.count -eq 0) {
|
||||
Main @("auditbeat","winlogbeat")
|
||||
} else {
|
||||
Main $args
|
||||
}
|
||||
94
Vagrant/resources/malcolm/scripts/beats/windows_vm_example/Vagrantfile
vendored
Normal file
94
Vagrant/resources/malcolm/scripts/beats/windows_vm_example/Vagrantfile
vendored
Normal file
@@ -0,0 +1,94 @@
|
||||
# -*- mode: ruby -*-
|
||||
# vi: set ft=ruby :
|
||||
|
||||
unless Vagrant.has_plugin?("vagrant-reload")
|
||||
raise 'vagrant-reload plugin is not installed!'
|
||||
end
|
||||
|
||||
# hack: https://github.com/hashicorp/vagrant/issues/8878#issuecomment-345112810
|
||||
class VagrantPlugins::ProviderVirtualBox::Action::Network
|
||||
def dhcp_server_matches_config?(dhcp_server, config)
|
||||
true
|
||||
end
|
||||
end
|
||||
|
||||
Vagrant.configure("2") do |config|
|
||||
|
||||
config.vm.box = "StefanScherer/windows_10"
|
||||
|
||||
config.vm.network "private_network", type: "dhcp"
|
||||
|
||||
config.vm.synced_folder ".", "c:/vagrant_shared", disabled: true
|
||||
|
||||
if Vagrant.has_plugin?("vagrant-vbguest")
|
||||
config.vbguest.auto_update = false
|
||||
end
|
||||
|
||||
config.vm.communicator = "winrm"
|
||||
|
||||
config.winrm.username = "vagrant"
|
||||
config.winrm.password = "vagrant"
|
||||
|
||||
config.vm.guest = :windows
|
||||
config.windows.halt_timeout = 15
|
||||
|
||||
# port forward WinRM and RDP
|
||||
config.vm.network :forwarded_port, guest: 3389, host: 3389, id: "rdp", auto_correct: true
|
||||
config.vm.network :forwarded_port, guest: 5985, host: 5985, id: "winrm", auto_correct: true
|
||||
|
||||
config.vm.provider :virtualbox do |vb, override|
|
||||
vb.gui = true
|
||||
vb.customize ["modifyvm", :id, "--memory", 4096]
|
||||
vb.customize ["modifyvm", :id, "--cpus", 2]
|
||||
vb.customize ["modifyvm", :id, "--vram", 256]
|
||||
vb.customize ["modifyvm", :id, "--ioapic", "on"]
|
||||
vb.customize ["modifyvm", :id, "--nestedpaging", "on"]
|
||||
vb.customize ["modifyvm", :id, "--pae", "on"]
|
||||
vb.customize ["modifyvm", :id, "--hwvirtex", "on"]
|
||||
vb.customize ["modifyvm", :id, "--nested-hw-virt", "on"]
|
||||
vb.customize ["modifyvm", :id, "--graphicscontroller", "vboxsvga"]
|
||||
vb.customize ["modifyvm", :id, "--accelerate2dvideo", "on"]
|
||||
vb.customize ["modifyvm", :id, "--accelerate3d", "on"]
|
||||
vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]
|
||||
vb.customize ["setextradata", "global", "GUI/SuppressMessages", "all" ]
|
||||
vb.customize ["modifyvm", :id, "--usb", "on"]
|
||||
vb.customize ["modifyvm", :id, "--usbehci", "on"]
|
||||
vb.customize ["modifyvm", :id, "--audio", "pulse", "--audiocontroller", "hda"]
|
||||
end
|
||||
|
||||
config.vm.provision "shell", inline: <<-STEP1
|
||||
New-Item -Path 'HKLM:\\SOFTWARE\\Policies\\Microsoft\\Windows' -Name CloudContent | Out-Null
|
||||
New-ItemProperty -Path 'HKLM:\\SOFTWARE\\Policies\\Microsoft\\Windows\\CloudContent' -Name 'DisableWindowsConsumerFeatures' -PropertyType DWORD -Value '1' -Force | Out-Null
|
||||
New-Item -Path 'HKLM:\\SOFTWARE\\Policies\\Microsoft\\Windows\\' -Name 'Windows Search' | Out-Null
|
||||
New-ItemProperty -Path 'HKLM:\\SOFTWARE\\Policies\\Microsoft\\Windows\\Windows Search' -Name 'AllowCortana' -PropertyType DWORD -Value '0' -Force | Out-Null
|
||||
Set-ItemProperty 'HKLM:\\SOFTWARE\\Microsoft\\SQMClient\\Windows' CEIPEnable 0 | Out-Null
|
||||
schtasks /Change /TN 'Microsoft\\Windows\\Customer Experience Improvement Program\\UsbCeip' /Disable | Out-Null
|
||||
|
||||
taskkill /f /im OneDrive.exe
|
||||
C:/Windows/SysWOW64/OneDriveSetup.exe /uninstall
|
||||
STEP1
|
||||
config.vm.provision :reload
|
||||
|
||||
config.vm.provision "shell", inline: <<-STEP2
|
||||
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
|
||||
choco install -y chocolateygui 7zip.install conemu firefox hackfont putty.install python sublimetext3 sysinternals
|
||||
|
||||
$beats = @("auditbeat","winlogbeat","packetbeat","metricbeat")
|
||||
foreach ($beat in $beats) {
|
||||
Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/$beat/$beat-oss-7.6.2-windows-x86_64.zip -OutFile $beat-7.6.2-windows-x86_64.zip -UseBasicParsing
|
||||
Expand-Archive -LiteralPath $beat-7.6.2-windows-x86_64.zip -DestinationPath 'C:\\Program Files'
|
||||
Remove-Item $beat-7.6.2-windows-x86_64.zip
|
||||
Rename-Item "C:\\Program Files\\$beat-7.6.2-windows-x86_64" "C:\\Program Files\\$beat"
|
||||
((Get-Content -path "C:\\Program Files\\$beat\\install-service-$beat.ps1" -Raw) -replace 'ProgramData','Program Files') | Set-Content -Path "C:\\Program Files\\$beat\\install-service-$beat.ps1"
|
||||
((Get-Content -path "C:\\Program Files\\$beat\\install-service-$beat.ps1" -Raw) -replace ' -path',' --path') | Set-Content -Path "C:\\Program Files\\$beat\\install-service-$beat.ps1"
|
||||
}
|
||||
STEP2
|
||||
|
||||
["auditbeat","winlogbeat","packetbeat","metricbeat"].to_enum.with_index(1).each do |beat, i|
|
||||
config.vm.provision "file", source: "./#{beat}.yml", destination: "C:\\Program Files\\#{beat}\\#{beat}.yml"
|
||||
config.vm.provision "file", source: "../beat_run.py", destination: "C:\\Program Files\\#{beat}\\beat_run.py"
|
||||
config.vm.provision "file", source: "../beat_config.py", destination: "C:\\Program Files\\#{beat}\\beat_config.py"
|
||||
config.vm.provision "file", source: "../beat_common.py", destination: "C:\\Program Files\\#{beat}\\beat_common.py"
|
||||
end
|
||||
end
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# See https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-reference-yml.html
|
||||
|
||||
# Thanks to "The Windows File Auditing Logging Cheat Sheet" at
|
||||
# https://www.malwarearchaeology.com/cheat-sheets
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
auditbeat.modules:
|
||||
|
||||
#------------------------------- file_integrity Module -----------------------
|
||||
|
||||
- module: file_integrity
|
||||
paths:
|
||||
- C:/Program Files
|
||||
- C:/Program Files/Internet Explorer
|
||||
- C:/Program Files/Common Files
|
||||
- C:/Program Files (x86)
|
||||
- C:/Program Files (x86) /Common Files
|
||||
- C:/ProgramData
|
||||
- C:/Windows
|
||||
- C:/Windows/System32
|
||||
- C:/Windows/System32/Drivers
|
||||
- C:/Windows/System32/Drivers/etc
|
||||
- C:/Windows/System32/Sysprep
|
||||
- C:/Windows/System32/wbem
|
||||
- C:/Windows/System32/WindowsPowerShell/v1.0
|
||||
- C:/Windows/Web
|
||||
- C:/Windows/SysWOW64
|
||||
- C:/Windows/SysWOW64/Drivers
|
||||
- C:/Windows/SysWOW64/wbem
|
||||
- C:/Windows/SysWOW64/WindowsPowerShell/v1.0
|
||||
recursive: false
|
||||
|
||||
- module: file_integrity
|
||||
paths:
|
||||
- C:/Boot
|
||||
- C:/Perflogs
|
||||
- C:/Users/All Users/Microsoft/Windows/Start Menu/Programs/Startup
|
||||
- C:/Users/Public
|
||||
# todo: wildcards handled?
|
||||
# - C:/Users/*/AppData/Local
|
||||
# - C:/Users/*/AppData/Local/Temp
|
||||
# - C:/Users/*/AppData/LocalLow
|
||||
# - C:/Users/*/AppData/Roaming
|
||||
- C:/Windows/Scripts
|
||||
- C:/Windows/System
|
||||
- C:/Windows/System32/GroupPolicy/Machine/Scripts/Startup
|
||||
- C:/Windows/System32/GroupPolicy/Machine/Scripts/Shutdown
|
||||
- C:/Windows/System32/GroupPolicy/User/Scripts/Logon
|
||||
- C:/Windows/System32/GroupPolicy/User/Scripts/Logoff
|
||||
- C:/Windows/System32/Repl
|
||||
recursive: true
|
||||
|
||||
# examples for exclusions if things are noisy
|
||||
# exclude_files:
|
||||
# - '(?i)\.blf$'
|
||||
# - '(?i)\.dat$'
|
||||
# - '(?i)\.lnk$'
|
||||
# - '(?i)\.log\w*$'
|
||||
# - '(?i)\.mum$'
|
||||
# - '(?i)\.regtrans-ms$'
|
||||
# - '(?i)\.swp$'
|
||||
# - '(?i)\.tmp$'
|
||||
# - '(?i)beat\.(lock|yml(\.new)?|db)$'
|
||||
# - '(?i)\\(assembly|CatRoot|CbsTemp|databases?|Deleted|diagnostics?|Log(File)?s?|Notifications?|Packages?|Prefetch|schemas?|servicing|Sessions?|SleepStudy|SoftwareDistribution|Tasks?|Temp|tracing|wbem|WinMetadata|WinSAT|WinSxS)\\?'
|
||||
# - '(?i)cache'
|
||||
|
||||
# TODO: system module is apparently only available in the non-OSS basic license :-(
|
||||
|
||||
# - module: system
|
||||
# datasets:
|
||||
# - host # General host information, e.g. uptime, IPs
|
||||
# period: 1m
|
||||
# state.period: 1h
|
||||
|
||||
# - module: system
|
||||
# datasets:
|
||||
# - process # Started and stopped processes
|
||||
# period: 1s
|
||||
@@ -0,0 +1,65 @@
|
||||
# See https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-reference-yml.html
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
metricbeat.config.modules:
|
||||
path: ${path.config}/conf.d/*.yml
|
||||
reload.period: 10s
|
||||
reload.enabled: false
|
||||
|
||||
metricbeat.max_start_delay: 10s
|
||||
|
||||
metricbeat.modules:
|
||||
|
||||
#------------------------------- System Module -------------------------------
|
||||
|
||||
- module: system
|
||||
period: 30s
|
||||
metricsets:
|
||||
- cpu # CPU usage
|
||||
- memory # Memory usage
|
||||
- network # Network IO
|
||||
- process # Per process metrics
|
||||
- process_summary # Process summary
|
||||
- uptime # System Uptime
|
||||
- diskio # Disk IO
|
||||
enabled: true
|
||||
processes: ['.*']
|
||||
process.include_top_n:
|
||||
enabled: true
|
||||
by_cpu: 10
|
||||
by_memory: 10
|
||||
|
||||
cpu.metrics: ["percentages"]
|
||||
core.metrics: ["percentages"]
|
||||
|
||||
- module: system
|
||||
period: 1m
|
||||
metricsets:
|
||||
- filesystem # File system usage for each mountpoint
|
||||
- fsstat # File system summary metrics
|
||||
enabled: true
|
||||
|
||||
- module: windows
|
||||
metricsets: ["perfmon"]
|
||||
enabled: true
|
||||
period: 10s
|
||||
perfmon.ignore_non_existent_counters: false
|
||||
perfmon.group_measurements_by_instance: true
|
||||
perfmon.queries:
|
||||
- object: "Process"
|
||||
instance: ["svchost*", "conhost*"]
|
||||
counters:
|
||||
- name: "% Processor Time"
|
||||
field: time.processor.pct
|
||||
format: "float"
|
||||
perfmon.counters:
|
||||
- instance_label: processor.name
|
||||
instance_name: total
|
||||
measurement_label: processor.time.total.pct
|
||||
query: '\Processor Information(_Total)\% Processor Time'
|
||||
|
||||
- module: windows
|
||||
metricsets: ["service"]
|
||||
enabled: true
|
||||
period: 60s
|
||||
@@ -0,0 +1,90 @@
|
||||
# See https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-reference-yml.html
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
# packetbeat.exe requires Npcap (https://nmap.org/npcap/#download) to be installed
|
||||
|
||||
|
||||
#------------------------------- network device ------------------------------
|
||||
|
||||
packetbeat.interfaces.device: 0
|
||||
packetbeat.interfaces.type: pcap
|
||||
packetbeat.interfaces.snaplen: 65535
|
||||
|
||||
#------------------------------- flows ---------------------------------------
|
||||
|
||||
packetbeat.flows:
|
||||
enabled: true
|
||||
timeout: 30s
|
||||
period: 10s
|
||||
|
||||
#------------------------------- transaction protocols -----------------------
|
||||
|
||||
packetbeat.protocols:
|
||||
- type: icmp
|
||||
enabled: true
|
||||
|
||||
- type: amqp
|
||||
enabled: true
|
||||
ports: [5672]
|
||||
|
||||
- type: cassandra
|
||||
enabled: true
|
||||
ports: [9042]
|
||||
|
||||
- type: dhcpv4
|
||||
enabled: true
|
||||
ports: [67, 68]
|
||||
|
||||
- type: dns
|
||||
enabled: true
|
||||
ports: [53]
|
||||
include_authorities: true
|
||||
include_additionals: true
|
||||
|
||||
- type: http
|
||||
enabled: true
|
||||
ports: [80, 8080, 8000, 5000, 8002]
|
||||
|
||||
- type: memcache
|
||||
enabled: true
|
||||
ports: [11211]
|
||||
|
||||
- type: mysql
|
||||
enabled: true
|
||||
ports: [3306,3307]
|
||||
|
||||
- type: pgsql
|
||||
enabled: true
|
||||
ports: [5432]
|
||||
|
||||
- type: redis
|
||||
enabled: true
|
||||
ports: [6379]
|
||||
|
||||
- type: thrift
|
||||
enabled: true
|
||||
ports: [9090]
|
||||
|
||||
- type: mongodb
|
||||
enabled: true
|
||||
ports: [27017]
|
||||
|
||||
- type: nfs
|
||||
enabled: true
|
||||
ports: [2049]
|
||||
|
||||
- type: tls
|
||||
enabled: true
|
||||
ports:
|
||||
- 443 # HTTPS
|
||||
- 993 # IMAPS
|
||||
- 995 # POP3S
|
||||
- 5223 # XMPP over SSL
|
||||
- 8883 # Secure MQTT
|
||||
- 9243 # Elasticsearch
|
||||
|
||||
#------------------------------- monitored processes -------------------------
|
||||
|
||||
packetbeat.procs.enabled: true
|
||||
packetbeat.ignore_outgoing: false
|
||||
@@ -0,0 +1,43 @@
|
||||
# see https://www.elastic.co/guide/en/beats/winlogbeat/master/winlogbeat-reference-yml.html
|
||||
|
||||
# also see some of these excellent cheat sheets for Windows logging:
|
||||
# https://www.malwarearchaeology.com/cheat-sheets
|
||||
|
||||
# ==============================================================================
|
||||
|
||||
#------------------------------- event logs ----------------------------------
|
||||
|
||||
winlogbeat.event_logs:
|
||||
- name: AMSI/Operational
|
||||
- name: Application
|
||||
ignore_older: 72h
|
||||
- name: ForwardedEvents
|
||||
tags: ["forwarded"]
|
||||
- name: Internet Explorer
|
||||
- name: Microsoft-Windows-LSA/Operational
|
||||
- name: Microsoft-Windows-PowerShell/Admin
|
||||
- name: Microsoft-Windows-PowerShell/Operational
|
||||
- name: Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Operational
|
||||
- name: Microsoft-Windows-Windows Defender/Operational
|
||||
- name: Microsoft-Windows-Windows Defender/WHC
|
||||
- name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall
|
||||
- name: Microsoft-Windows-WMI-Activity/Operational
|
||||
- name: OpenSSH/Admin
|
||||
- name: OpenSSH/Operational
|
||||
|
||||
# TODO: the Security and Sysmon modules are apparently only available in the non-OSS basic license :-(
|
||||
|
||||
# - name: Security
|
||||
# processors:
|
||||
# - script:
|
||||
# lang: javascript
|
||||
# id: security
|
||||
# file: ${path.home}/module/security/config/winlogbeat-security.js
|
||||
# - name: System
|
||||
# - name: Windows PowerShell
|
||||
# - name: Microsoft-Windows-Sysmon/Operational
|
||||
# processors:
|
||||
# - script:
|
||||
# lang: javascript
|
||||
# id: sysmon
|
||||
# file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
|
||||
119
Vagrant/resources/malcolm/scripts/build.sh
Executable file
119
Vagrant/resources/malcolm/scripts/build.sh
Executable file
@@ -0,0 +1,119 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
if [ -z "$BASH_VERSION" ]; then
|
||||
echo "Wrong interpreter, please run \"$0\" with bash"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$(uname -s)" = 'Darwin' ]] && REALPATH=grealpath || REALPATH=realpath
|
||||
[[ "$(uname -s)" = 'Darwin' ]] && DIRNAME=gdirname || DIRNAME=dirname
|
||||
[[ "$(uname -s)" = 'Darwin' ]] && GREP=ggrep || GREP=grep
|
||||
if ! (type "$REALPATH" && type "$DIRNAME" && type "$GREP") > /dev/null; then
|
||||
echo "$(basename "${BASH_SOURCE[0]}") requires $REALPATH and $DIRNAME and $GREP"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if docker-compose version >/dev/null 2>&1; then
|
||||
DOCKER_COMPOSE_BIN=docker-compose
|
||||
DOCKER_BIN=docker
|
||||
elif $GREP -q Microsoft /proc/version && docker-compose.exe version >/dev/null 2>&1; then
|
||||
DOCKER_COMPOSE_BIN=docker-compose.exe
|
||||
DOCKER_BIN=docker.exe
|
||||
fi
|
||||
|
||||
if [ "$1" ]; then
|
||||
CONFIG_FILE="$1"
|
||||
DOCKER_COMPOSE_COMMAND="$DOCKER_COMPOSE_BIN -f "$CONFIG_FILE""
|
||||
shift # use remainder of arguments for services
|
||||
else
|
||||
CONFIG_FILE="docker-compose.yml"
|
||||
DOCKER_COMPOSE_COMMAND="$DOCKER_COMPOSE_BIN"
|
||||
fi
|
||||
|
||||
function filesize_in_image() {
|
||||
FILESPEC="$2"
|
||||
IMAGE="$($GREP -P "^\s+image:.*$1" docker-compose-standalone.yml | awk '{print $2}')"
|
||||
$DOCKER_BIN run --rm --entrypoint /bin/sh "$IMAGE" -c "stat --printf='%s' \"$FILESPEC\" 2>/dev/null || stat -c '%s' \"$FILESPEC\" 2>/dev/null"
|
||||
}
|
||||
|
||||
# force-navigate to Malcolm base directory (parent of scripts/ directory)
|
||||
SCRIPT_PATH="$($DIRNAME $($REALPATH -e "${BASH_SOURCE[0]}"))"
|
||||
pushd "$SCRIPT_PATH/.." >/dev/null 2>&1
|
||||
|
||||
# make sure docker is installed, at this point it's required
|
||||
if ! $DOCKER_BIN info >/dev/null 2>&1 ; then
|
||||
echo "Docker is not installed, or not runable as this user."
|
||||
echo "Install docker (install.py may help with that) and try again later."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# make sure docker-compose is installed, at this point it's required
|
||||
if ! $DOCKER_COMPOSE_BIN version >/dev/null 2>&1 ; then
|
||||
echo "Docker Compose is not installed, or not runable as this user."
|
||||
echo "Install docker-compose (install.py may help with that) and try again later."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
unset CONFIRMATION
|
||||
read -p "Malcolm Docker images will now be built and/or pulled, force full clean (non-cached) rebuild [y/N]? " CONFIRMATION
|
||||
CONFIRMATION=${CONFIRMATION:-N}
|
||||
|
||||
BUILD_DATE="$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
|
||||
MALCOLM_VERSION="$($GREP -P "^\s+image:\s*malcolm" "$CONFIG_FILE" | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)"
|
||||
VCS_REVISION="$(git rev-parse --short HEAD 2>/dev/null || echo unknown)"
|
||||
|
||||
[[ ! -f ./auth.env ]] && touch ./auth.env
|
||||
|
||||
# MaxMind now requires a (free) license key to download the free versions of their GeoIP databases.
|
||||
if [ ${#MAXMIND_GEOIP_DB_LICENSE_KEY} -gt 1 ]; then
|
||||
# prefer a local environment variable
|
||||
MAXMIND_API_KEY="$MAXMIND_GEOIP_DB_LICENSE_KEY"
|
||||
else
|
||||
# but default to what they have saved in the docker-compose YML file
|
||||
MAXMIND_API_KEY="$($GREP -P "^\s*MAXMIND_GEOIP_DB_LICENSE_KEY\s*:\s" "$CONFIG_FILE" | cut -d: -f2 | tr -d '[:space:]'\'\" | head -n 1)"
|
||||
fi
|
||||
|
||||
# for some debug branches this may be used to download artifacts from github
|
||||
if [ ${#GITHUB_OAUTH_TOKEN} -gt 1 ]; then
|
||||
# prefer a local environment variable
|
||||
GITHUB_TOKEN="$GITHUB_OAUTH_TOKEN"
|
||||
else
|
||||
# nope
|
||||
GITHUB_TOKEN="0"
|
||||
fi
|
||||
|
||||
if [[ $CONFIRMATION =~ ^[Yy] ]]; then
|
||||
$DOCKER_COMPOSE_COMMAND build --force-rm --no-cache --build-arg MAXMIND_GEOIP_DB_LICENSE_KEY="$MAXMIND_API_KEY" --build-arg GITHUB_OAUTH_TOKEN="$GITHUB_TOKEN" --build-arg BUILD_DATE="$BUILD_DATE" --build-arg MALCOLM_VERSION="$MALCOLM_VERSION" --build-arg VCS_REVISION="$VCS_REVISION" "$@"
|
||||
else
|
||||
$DOCKER_COMPOSE_COMMAND build --build-arg MAXMIND_GEOIP_DB_LICENSE_KEY="$MAXMIND_API_KEY" --build-arg GITHUB_OAUTH_TOKEN="$GITHUB_TOKEN" --build-arg BUILD_DATE="$BUILD_DATE" --build-arg MALCOLM_VERSION="$MALCOLM_VERSION" --build-arg VCS_REVISION="$VCS_REVISION" "$@"
|
||||
fi
|
||||
|
||||
# we're going to do some validation that some things got pulled/built correctly
|
||||
FILES_IN_IMAGES=(
|
||||
"/usr/share/filebeat/filebeat.yml;filebeat-oss"
|
||||
"/var/lib/clamav/main.cvd;file-monitor"
|
||||
"/var/lib/clamav/daily.cvd;file-monitor"
|
||||
"/var/lib/clamav/bytecode.cvd;file-monitor"
|
||||
"/var/www/upload/js/jquery.fileupload.js;file-upload"
|
||||
"/opt/freq_server/freq_server.py;freq"
|
||||
"/var/www/htadmin/index.php;htadmin"
|
||||
"/usr/share/logstash/config/oui-logstash.txt;logstash"
|
||||
"/etc/ip_protocol_numbers.yaml;logstash"
|
||||
"/etc/ja3.yaml;logstash"
|
||||
"/data/moloch/etc/GeoLite2-ASN.mmdb;arkime"
|
||||
"/data/moloch/etc/GeoLite2-Country.mmdb;arkime"
|
||||
"/data/moloch/etc/ipv4-address-space.csv;arkime"
|
||||
"/data/moloch/etc/oui.txt;arkime"
|
||||
"/data/moloch/bin/moloch-capture;arkime"
|
||||
"/var/www/html/list.min.js;name-map-ui"
|
||||
"/var/www/html/jquery.min.js;name-map-ui"
|
||||
"/opt/zeek/bin/zeek;zeek"
|
||||
"/opt/spicy/lib/libspicy.so;zeek"
|
||||
)
|
||||
for i in ${FILES_IN_IMAGES[@]}; do
|
||||
FILE="$(echo "$i" | cut -d';' -f1)"
|
||||
IMAGE="$(echo "$i" | cut -d';' -f2)"
|
||||
(( "$(filesize_in_image $IMAGE "$FILE")" > 0 )) || { echo "Failed to create \"$FILE\" in \"$IMAGE\""; exit 1; }
|
||||
done
|
||||
886
Vagrant/resources/malcolm/scripts/control.py
Executable file
886
Vagrant/resources/malcolm/scripts/control.py
Executable file
@@ -0,0 +1,886 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
import argparse
|
||||
import errno
|
||||
import getpass
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import sys
|
||||
|
||||
from malcolm_common import *
|
||||
from collections import defaultdict, namedtuple
|
||||
from subprocess import (PIPE, STDOUT, Popen, check_call, CalledProcessError)
|
||||
try:
|
||||
from contextlib import nullcontext
|
||||
except ImportError:
|
||||
class nullcontext(object):
|
||||
def __init__(self, enter_result=None):
|
||||
self.enter_result = enter_result
|
||||
def __enter__(self):
|
||||
return self.enter_result
|
||||
def __exit__(self, *args):
|
||||
pass
|
||||
|
||||
###################################################################################################
|
||||
ScriptName = os.path.basename(__file__)
|
||||
|
||||
pyPlatform = platform.system()
|
||||
|
||||
args = None
|
||||
dockerBin = None
|
||||
dockerComposeBin = None
|
||||
opensslBin = None
|
||||
|
||||
###################################################################################################
|
||||
try:
|
||||
from colorama import init as ColoramaInit, Fore, Back, Style
|
||||
ColoramaInit()
|
||||
coloramaImported = True
|
||||
except:
|
||||
coloramaImported = False
|
||||
|
||||
###################################################################################################
|
||||
# perform a service-keystore operation in a Docker container
|
||||
#
|
||||
# service - the service in the docker-compose YML file
|
||||
# keystore_args - arguments to pass to the service-keystore binary in the container
|
||||
# run_process_kwargs - keyword arguments to pass to run_process
|
||||
#
|
||||
# returns True (success) or False (failure)
|
||||
#
|
||||
def keystore_op(service, dropPriv=False, *keystore_args, **run_process_kwargs):
|
||||
global args
|
||||
global dockerBin
|
||||
global dockerComposeBin
|
||||
|
||||
err = -1
|
||||
results = []
|
||||
|
||||
# the elastic containers all follow the same naming pattern for these executables
|
||||
keystoreBinProc = f"/usr/share/{service}/bin/{service}-keystore"
|
||||
|
||||
# if we're using docker-uid-gid-setup.sh to drop privileges as we spin up a container
|
||||
dockerUidGuidSetup = "/usr/local/bin/docker-uid-gid-setup.sh"
|
||||
|
||||
# docker-compose use local temporary path
|
||||
osEnv = os.environ.copy()
|
||||
osEnv['TMPDIR'] = MalcolmTmpPath
|
||||
|
||||
# open up the docker-compose file and "grep" for the line where the keystore file
|
||||
# is bind-mounted into the service container (once and only once). the bind
|
||||
# mount needs to exist in the YML file and the local directory containing the
|
||||
# keystore file needs to exist (although the file itself might not yet).
|
||||
# also get PUID and PGID variables from the docker-compose file.
|
||||
localKeystore = None
|
||||
localKeystoreDir = None
|
||||
localKeystorePreExists = False
|
||||
volumeKeystore = None
|
||||
volumeKeystoreDir = None
|
||||
uidGidDict = defaultdict(str)
|
||||
|
||||
try:
|
||||
|
||||
composeFileLines = list()
|
||||
uidGidDict['PUID'] = f'{os.getuid()}' if (pyPlatform != PLATFORM_WINDOWS) else '1000'
|
||||
uidGidDict['PGID'] = f'{os.getgid()}' if (pyPlatform != PLATFORM_WINDOWS) else '1000'
|
||||
with open(args.composeFile, 'r') as f:
|
||||
allLines = f.readlines()
|
||||
composeFileLines = [x for x in allLines if re.search(fr'-.*?{service}.keystore\s*:.*{service}.keystore', x)]
|
||||
uidGidDict.update(dict(x.split(':') for x in [''.join(x.split()) for x in allLines if re.search(fr'^\s*P[UG]ID\s*:\s*\d+\s*$', x)]))
|
||||
|
||||
if (len(composeFileLines) == 1) and (len(composeFileLines[0]) > 0):
|
||||
matches = re.search(fr'-\s*(?P<localKeystore>.*?{service}.keystore)\s*:\s*(?P<volumeKeystore>.*?{service}.keystore)', composeFileLines[0])
|
||||
if matches:
|
||||
localKeystore = os.path.realpath(matches.group('localKeystore'))
|
||||
localKeystoreDir = os.path.dirname(localKeystore)
|
||||
volumeKeystore = matches.group('volumeKeystore')
|
||||
volumeKeystoreDir = os.path.dirname(volumeKeystore)
|
||||
|
||||
if (localKeystore is not None) and (volumeKeystore is not None) and os.path.isdir(localKeystoreDir):
|
||||
localKeystorePreExists = os.path.isfile(localKeystore)
|
||||
|
||||
dockerCmd = None
|
||||
|
||||
# determine if Malcolm is running; if so, we'll use docker-compose exec, other wise we'll use docker run
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'ps', '-q', service], env=osEnv, debug=args.debug)
|
||||
out[:] = [x for x in out if x]
|
||||
if (err == 0) and (len(out) > 0):
|
||||
# Malcolm is running, we can use an existing container
|
||||
|
||||
# assemble the service-keystore command
|
||||
dockerCmd = [dockerComposeBin, 'exec',
|
||||
|
||||
# if using stdin, indicate the container is "interactive", else noop (duplicate --rm)
|
||||
'-T' if ('stdin' in run_process_kwargs and run_process_kwargs['stdin']) else '',
|
||||
|
||||
# execute as UID:GID in docker-compose.yml file
|
||||
'-u', f'{uidGidDict["PUID"]}:{uidGidDict["PGID"]}'
|
||||
|
||||
# the work directory in the container is the directory to contain the keystore file
|
||||
'-w', volumeKeystoreDir,
|
||||
|
||||
# the service name
|
||||
service,
|
||||
|
||||
# the executable filespec
|
||||
keystoreBinProc]
|
||||
|
||||
else:
|
||||
# Malcolm isn't running, do 'docker run' to spin up a temporary container to run the ocmmand
|
||||
|
||||
# "grep" the docker image out of the service's image: value from the docker-compose YML file
|
||||
serviceImage = None
|
||||
composeFileLines = list()
|
||||
with open(args.composeFile, 'r') as f:
|
||||
composeFileLines = [x for x in f.readlines() if f'image: malcolmnetsec/{service}' in x]
|
||||
if (len(composeFileLines) > 0) and (len(composeFileLines[0]) > 0):
|
||||
imageLineValues = composeFileLines[0].split()
|
||||
if (len(imageLineValues) > 1):
|
||||
serviceImage = imageLineValues[1]
|
||||
|
||||
if serviceImage is not None:
|
||||
# assemble the service-keystore command
|
||||
dockerCmd = [dockerBin, 'run',
|
||||
|
||||
# remove the container when complete
|
||||
'--rm',
|
||||
|
||||
# if using stdin, indicate the container is "interactive", else noop
|
||||
'-i' if ('stdin' in run_process_kwargs and run_process_kwargs['stdin']) else '',
|
||||
|
||||
# if dropPriv, dockerUidGuidSetup will take care of dropping privileges for the correct UID/GID
|
||||
# if NOT dropPriv, enter with the keystore executable directly
|
||||
'--entrypoint', dockerUidGuidSetup if dropPriv else keystoreBinProc,
|
||||
'--env', f'DEFAULT_UID={uidGidDict["PUID"]}',
|
||||
'--env', f'DEFAULT_GID={uidGidDict["PGID"]}',
|
||||
'--env', f'PUSER_CHOWN={volumeKeystoreDir}',
|
||||
|
||||
# rw bind mount the local directory to contain the keystore file to the container directory
|
||||
'-v', f'{localKeystoreDir}:{volumeKeystoreDir}:rw',
|
||||
|
||||
# the work directory in the container is the directory to contain the keystore file
|
||||
'-w', volumeKeystoreDir,
|
||||
|
||||
# if dropPriv, execute as root, as docker-uid-gid-setup.sh will drop privileges for us
|
||||
# if NOT dropPriv, execute as UID:GID in docker-compose.yml file
|
||||
'-u', 'root' if dropPriv else f'{uidGidDict["PUID"]}:{uidGidDict["PGID"]}',
|
||||
|
||||
# the service image name grepped from the YML file
|
||||
serviceImage]
|
||||
|
||||
if dropPriv:
|
||||
# the keystore executable filespec (as we used dockerUidGuidSetup as the entrypoint)
|
||||
dockerCmd.append(keystoreBinProc)
|
||||
|
||||
else:
|
||||
raise Exception(f'Unable to identify docker image for {service} in {args.composeFile}')
|
||||
|
||||
if (dockerCmd is not None):
|
||||
|
||||
# append whatever other arguments to pass to the executable filespec
|
||||
if keystore_args:
|
||||
dockerCmd.extend(list(keystore_args))
|
||||
|
||||
dockerCmd[:] = [x for x in dockerCmd if x]
|
||||
|
||||
# execute the command, passing through run_process_kwargs to run_process as expanded keyword arguments
|
||||
err, results = run_process(dockerCmd, env=osEnv, debug=args.debug, **run_process_kwargs)
|
||||
if (err != 0) or (not os.path.isfile(localKeystore)):
|
||||
raise Exception(f'Error processing command {service} keystore: {results}')
|
||||
|
||||
else:
|
||||
raise Exception(f'Unable formulate keystore command for {service} in {args.composeFile}')
|
||||
|
||||
else:
|
||||
raise Exception(f'Unable to identify a unique keystore file bind mount for {service} in {args.composeFile}')
|
||||
|
||||
except Exception as e:
|
||||
if (err == 0):
|
||||
err = -1
|
||||
|
||||
# don't be so whiny if the "create" failed just because it already existed or a 'remove' failed on a nonexistant item
|
||||
if ((not args.debug) and
|
||||
list(keystore_args) and
|
||||
(len(list(keystore_args)) > 0) and
|
||||
(list(keystore_args)[0].lower() in ('create', 'remove')) and
|
||||
localKeystorePreExists):
|
||||
pass
|
||||
else:
|
||||
eprint(e)
|
||||
|
||||
# success = (error == 0)
|
||||
return (err == 0), results
|
||||
|
||||
###################################################################################################
|
||||
def status():
|
||||
global args
|
||||
global dockerComposeBin
|
||||
|
||||
# docker-compose use local temporary path
|
||||
osEnv = os.environ.copy()
|
||||
osEnv['TMPDIR'] = MalcolmTmpPath
|
||||
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'ps'], env=osEnv, debug=args.debug)
|
||||
if (err == 0):
|
||||
print("\n".join(out))
|
||||
else:
|
||||
eprint("Failed to display Malcolm status\n")
|
||||
eprint("\n".join(out))
|
||||
exit(err)
|
||||
|
||||
###################################################################################################
|
||||
def logs():
|
||||
global args
|
||||
global dockerBin
|
||||
global dockerComposeBin
|
||||
|
||||
# noisy logs (a lot of it is NGINX logs from health checks)
|
||||
ignoreRegEx = re.compile(r"""
|
||||
.+(
|
||||
deprecated
|
||||
| DEPRECATION
|
||||
| eshealth
|
||||
| remov(ed|ing)\s+(old\s+file|dead\s+symlink|empty\s+directory)
|
||||
| update_mapping
|
||||
| throttling\s+index
|
||||
| executing\s+attempt_(transition|set_replica_count)\s+for
|
||||
| but\s+there\s+are\s+no\s+living\s+connections
|
||||
| saved_objects
|
||||
| /kibana/(api/ui_metric/report|internal/search/es)
|
||||
| retry\.go.+(send\s+unwait|done$)
|
||||
| scheduling\s+job\s*id.+opendistro-ism
|
||||
| descheduling\s+job\s*id
|
||||
| updating\s+number_of_replicas
|
||||
| running\s+full\s+sweep
|
||||
| (async|output)\.go.+(reset\s+by\s+peer|Connecting\s+to\s+backoff|backoff.+established$)
|
||||
| \b(d|es)?stats\.json
|
||||
| /_ns_/nstest\.html
|
||||
| esindices/list
|
||||
| _cat/indices
|
||||
| use_field_mapping
|
||||
| reaped\s+unknown\s+pid
|
||||
| Successfully\s+handled\s+GET\s+request\s+for\s+'/'
|
||||
| GET\s+/(_cat/health|api/status|sessions2-).+HTTP/[\d\.].+\b200\b
|
||||
| POST\s+/(d?stats/(d?stat|_doc|_search)|_bulk|fields/(field/)?_search).+HTTP/[\d\.].+\b20[01]\b
|
||||
| POST\s+HTTP/[\d\.].+\b200\b
|
||||
| POST\s+/server/php/\s+HTTP/\d+\.\d+"\s+\d+\s+\d+.*:8443/
|
||||
| curl.+localhost.+GET\s+/api/status\s+200
|
||||
| "GET\s+/\s+HTTP/1\.\d+"\s+200\s+-
|
||||
| \b1.+GET\s+/\s+.+401.+curl
|
||||
)
|
||||
""", re.VERBOSE | re.IGNORECASE)
|
||||
|
||||
serviceRegEx = re.compile(r'^(?P<service>.+?\|)\s*(?P<message>.*)$')
|
||||
|
||||
# increase COMPOSE_HTTP_TIMEOUT to be ridiculously large so docker-compose never times out the TTY doing debug output
|
||||
osEnv = os.environ.copy()
|
||||
osEnv['COMPOSE_HTTP_TIMEOUT'] = '100000000'
|
||||
# docker-compose use local temporary path
|
||||
osEnv['TMPDIR'] = MalcolmTmpPath
|
||||
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'ps'], env=osEnv, debug=args.debug)
|
||||
print("\n".join(out))
|
||||
|
||||
process = Popen([dockerComposeBin, '-f', args.composeFile, 'logs', '-f'], env=osEnv, stdout=PIPE)
|
||||
while True:
|
||||
output = process.stdout.readline()
|
||||
if (len(output) == 0) and (process.poll() is not None):
|
||||
break
|
||||
if output:
|
||||
outputStr = output.decode().strip()
|
||||
outputStrEscaped = EscapeAnsi(outputStr)
|
||||
if ignoreRegEx.match(outputStrEscaped):
|
||||
pass ### print(f'!!!!!!!: {outputStr}')
|
||||
else:
|
||||
serviceMatch = serviceRegEx.search(outputStrEscaped)
|
||||
serviceMatchFmt = serviceRegEx.search(outputStr) if coloramaImported else serviceMatch
|
||||
serviceStr = serviceMatchFmt.group('service') if (serviceMatchFmt is not None) else ''
|
||||
messageStr = serviceMatch.group('message') if (serviceMatch is not None) else ''
|
||||
outputJson = LoadStrIfJson(messageStr)
|
||||
if (outputJson is not None):
|
||||
|
||||
# if there's a timestamp in the JSON, move it outside of the JSON to the beginning of the log string
|
||||
timeKey = None
|
||||
if 'time' in outputJson:
|
||||
timeKey = 'time'
|
||||
elif 'timestamp' in outputJson:
|
||||
timeKey = 'timestamp'
|
||||
elif '@timestamp' in outputJson:
|
||||
timeKey = '@timestamp'
|
||||
timeStr = ''
|
||||
if timeKey is not None:
|
||||
timeStr = f"{outputJson[timeKey]} "
|
||||
outputJson.pop(timeKey, None)
|
||||
|
||||
if ('job.schedule' in outputJson) and ('job.position' in outputJson) and ('job.command' in outputJson):
|
||||
|
||||
# this is a status output line from supercronic, let's format and clean it up so it fits in better with the rest of the logs
|
||||
|
||||
# remove some clutter for the display
|
||||
for noisyKey in ['level', 'channel', 'iteration', 'job.position', 'job.schedule']:
|
||||
outputJson.pop(noisyKey, None)
|
||||
|
||||
# if it's just command and message, format those NOT as JSON
|
||||
jobCmd = outputJson['job.command']
|
||||
jobStatus = outputJson['msg']
|
||||
if (len(outputJson.keys()) == 2) and ('job.command' in outputJson) and ('msg' in outputJson):
|
||||
# if it's the most common status (starting or job succeeded) then don't print unless debug mode
|
||||
if args.debug or ((jobStatus != 'starting') and (jobStatus != 'job succeeded')):
|
||||
print(f"{serviceStr}{Style.RESET_ALL if coloramaImported else ''} {timeStr} {jobCmd}: {jobStatus}")
|
||||
else:
|
||||
pass
|
||||
|
||||
else:
|
||||
# standardize and print the JSON output
|
||||
print(f"{serviceStr}{Style.RESET_ALL if coloramaImported else ''} {timeStr}{json.dumps(outputJson)}")
|
||||
|
||||
elif ('kibana' in serviceStr):
|
||||
# this is an output line from kibana, let's clean it up a bit: remove some clutter for the display
|
||||
for noisyKey in ['type', 'tags', 'pid', 'method', 'prevState', 'prevMsg']:
|
||||
outputJson.pop(noisyKey, None)
|
||||
|
||||
# standardize and print the JSON output
|
||||
print(f"{serviceStr}{Style.RESET_ALL if coloramaImported else ''} {timeStr}{json.dumps(outputJson)}")
|
||||
|
||||
else:
|
||||
# standardize and print the JSON output
|
||||
print(f"{serviceStr}{Style.RESET_ALL if coloramaImported else ''} {timeStr}{json.dumps(outputJson)}")
|
||||
|
||||
else:
|
||||
# just a regular non-JSON string, print as-is
|
||||
print(outputStr if coloramaImported else outputStrEscaped)
|
||||
|
||||
else:
|
||||
time.sleep(0.5)
|
||||
process.poll()
|
||||
|
||||
###################################################################################################
|
||||
def stop(wipe=False):
|
||||
global args
|
||||
global dockerBin
|
||||
global dockerComposeBin
|
||||
|
||||
# docker-compose use local temporary path
|
||||
osEnv = os.environ.copy()
|
||||
osEnv['TMPDIR'] = MalcolmTmpPath
|
||||
|
||||
if wipe:
|
||||
# attempt to DELETE _template/zeek_template in Elasticsearch
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'exec', 'arkime', 'bash', '-c', 'curl -fs --output /dev/null -H"Content-Type: application/json" -XDELETE "http://$ES_HOST:$ES_PORT/_template/zeek_template"'], env=osEnv, debug=args.debug)
|
||||
|
||||
# if stop.sh is being called with wipe.sh (after the docker-compose file)
|
||||
# then also remove named and anonymous volumes (not external volumes, of course)
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'down', '--volumes'][:5 if wipe else -1], env=osEnv, debug=args.debug)
|
||||
if (err == 0):
|
||||
eprint("Stopped Malcolm\n")
|
||||
else:
|
||||
eprint("Malcolm failed to stop\n")
|
||||
eprint("\n".join(out))
|
||||
exit(err)
|
||||
|
||||
if wipe:
|
||||
# delete elasticsearch database
|
||||
shutil.rmtree(os.path.join(MalcolmPath, 'elasticsearch/nodes'), ignore_errors=True)
|
||||
|
||||
# delete data files (backups, zeek logs, arkime logs, PCAP files, captured PCAP files)
|
||||
for dataDir in ['elasticsearch-backup', 'zeek-logs', 'moloch-logs', 'pcap', 'moloch-raw']:
|
||||
for root, dirnames, filenames in os.walk(os.path.join(MalcolmPath, dataDir), topdown=True, onerror=None):
|
||||
for file in filenames:
|
||||
fileSpec = os.path.join(root, file)
|
||||
if (os.path.isfile(fileSpec) or os.path.islink(fileSpec)) and (not file.startswith('.git')):
|
||||
try:
|
||||
os.remove(fileSpec)
|
||||
except:
|
||||
pass
|
||||
|
||||
# clean up empty directories
|
||||
for dataDir in [os.path.join('elasticsearch-backup', 'logs'), os.path.join('zeek-logs', 'processed'), os.path.join('zeek-logs', 'current')]:
|
||||
RemoveEmptyFolders(dataDir, removeRoot=False)
|
||||
|
||||
eprint("Malcolm has been stopped and its data cleared\n")
|
||||
|
||||
###################################################################################################
|
||||
def start():
|
||||
global args
|
||||
global dockerBin
|
||||
global dockerComposeBin
|
||||
|
||||
# make sure the auth files exist. if we are in an interactive shell and we're
|
||||
# missing any of the auth files, prompt to create them now
|
||||
if sys.__stdin__.isatty() and (not MalcolmAuthFilesExist()):
|
||||
authSetup()
|
||||
|
||||
# still missing? sorry charlie
|
||||
if (not MalcolmAuthFilesExist()):
|
||||
raise Exception('Malcolm administrator account authentication files are missing, please run ./scripts/auth_setup to generate them')
|
||||
|
||||
# touch the metadata file
|
||||
open(os.path.join(MalcolmPath, os.path.join('htadmin', 'metadata')), 'a').close()
|
||||
|
||||
# if the elasticsearch and logstash keystore don't exist exist, create empty ones
|
||||
if not os.path.isfile(os.path.join(MalcolmPath, os.path.join('elasticsearch', 'elasticsearch.keystore'))):
|
||||
keystore_op('elasticsearch', True, 'create')
|
||||
if not os.path.isfile(os.path.join(MalcolmPath, os.path.join('logstash', os.path.join('certs', 'logstash.keystore')))):
|
||||
keystore_op('logstash', True, 'create')
|
||||
|
||||
# make sure permissions are set correctly for the nginx worker processes
|
||||
for authFile in [os.path.join(MalcolmPath, os.path.join('nginx', 'htpasswd')),
|
||||
os.path.join(MalcolmPath, os.path.join('htadmin', 'config.ini')),
|
||||
os.path.join(MalcolmPath, os.path.join('htadmin', 'metadata'))]:
|
||||
# chmod 644 authFile
|
||||
os.chmod(authFile, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
|
||||
for authFile in [os.path.join(MalcolmPath, os.path.join('nginx', 'nginx_ldap.conf'))]:
|
||||
# chmod 600 authFile
|
||||
os.chmod(authFile, stat.S_IRUSR | stat.S_IWUSR)
|
||||
|
||||
# make sure some directories exist before we start
|
||||
for path in [os.path.join(MalcolmPath, 'elasticsearch'),
|
||||
os.path.join(MalcolmPath, 'elasticsearch-backup'),
|
||||
os.path.join(MalcolmPath, os.path.join('nginx', 'ca-trust')),
|
||||
os.path.join(MalcolmPath, os.path.join('pcap', 'upload')),
|
||||
os.path.join(MalcolmPath, os.path.join('pcap', 'processed')),
|
||||
os.path.join(MalcolmPath, os.path.join('zeek-logs', 'current')),
|
||||
os.path.join(MalcolmPath, os.path.join('zeek-logs', 'upload')),
|
||||
os.path.join(MalcolmPath, os.path.join('zeek-logs', 'processed')),
|
||||
os.path.join(MalcolmPath, os.path.join('zeek-logs', 'extract_files'))]:
|
||||
try:
|
||||
os.makedirs(path)
|
||||
except OSError as exc:
|
||||
if (exc.errno == errno.EEXIST) and os.path.isdir(path):
|
||||
pass
|
||||
else:
|
||||
raise
|
||||
|
||||
# increase COMPOSE_HTTP_TIMEOUT to be ridiculously large so docker-compose never times out the TTY doing debug output
|
||||
osEnv = os.environ.copy()
|
||||
osEnv['COMPOSE_HTTP_TIMEOUT'] = '100000000'
|
||||
# docker-compose use local temporary path
|
||||
osEnv['TMPDIR'] = MalcolmTmpPath
|
||||
|
||||
# start docker
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'up', '--detach'], env=osEnv, debug=args.debug)
|
||||
if (err == 0):
|
||||
eprint("Started Malcolm\n\n")
|
||||
eprint("In a few minutes, Malcolm services will be accessible via the following URLs:")
|
||||
eprint("------------------------------------------------------------------------------")
|
||||
eprint(" - Arkime: https://localhost/")
|
||||
eprint(" - Kibana: https://localhost/kibana/")
|
||||
eprint(" - PCAP upload (web): https://localhost/upload/")
|
||||
eprint(" - PCAP upload (sftp): sftp://username@127.0.0.1:8022/files/")
|
||||
eprint(" - Host and subnet name mapping editor: https://localhost/name-map-ui/\n")
|
||||
eprint(" - Account management: https://localhost:488/\n")
|
||||
else:
|
||||
eprint("Malcolm failed to start\n")
|
||||
eprint("\n".join(out))
|
||||
exit(err)
|
||||
|
||||
###################################################################################################
|
||||
def authSetup(wipe=False):
|
||||
global args
|
||||
global dockerBin
|
||||
global dockerComposeBin
|
||||
global opensslBin
|
||||
|
||||
if YesOrNo('Store administrator username/password for local Malcolm access?', default=True):
|
||||
|
||||
# prompt username and password
|
||||
usernamePrevious = None
|
||||
password = None
|
||||
passwordConfirm = None
|
||||
passwordEncrypted = ''
|
||||
username = AskForString("Administrator username")
|
||||
|
||||
while True:
|
||||
password = AskForPassword(f"{username} password: ")
|
||||
passwordConfirm = AskForPassword(f"{username} password (again): ")
|
||||
if (password == passwordConfirm):
|
||||
break
|
||||
eprint("Passwords do not match")
|
||||
|
||||
# get previous admin username to remove from htpasswd file if it's changed
|
||||
authEnvFile = os.path.join(MalcolmPath, 'auth.env')
|
||||
if os.path.isfile(authEnvFile):
|
||||
prevAuthInfo = defaultdict(str)
|
||||
with open(authEnvFile, 'r') as f:
|
||||
for line in f:
|
||||
try:
|
||||
k, v = line.rstrip().split("=")
|
||||
prevAuthInfo[k] = v.strip('"')
|
||||
except:
|
||||
pass
|
||||
if (len(prevAuthInfo['MALCOLM_USERNAME']) > 0):
|
||||
usernamePrevious = prevAuthInfo['MALCOLM_USERNAME']
|
||||
|
||||
# get openssl hash of password
|
||||
err, out = run_process([opensslBin, 'passwd', '-1', '-stdin'], stdin=password, stderr=False, debug=args.debug)
|
||||
if (err == 0) and (len(out) > 0) and (len(out[0]) > 0):
|
||||
passwordEncrypted = out[0]
|
||||
else:
|
||||
raise Exception('Unable to generate password hash with openssl')
|
||||
|
||||
# write auth.env (used by htadmin and file-upload containers)
|
||||
with open(authEnvFile, 'w') as f:
|
||||
f.write("# Malcolm Administrator username and encrypted password for nginx reverse proxy (and upload server's SFTP access)\n")
|
||||
f.write(f'MALCOLM_USERNAME={username}\n')
|
||||
f.write(f'MALCOLM_PASSWORD={passwordEncrypted}\n')
|
||||
os.chmod(authEnvFile, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
|
||||
|
||||
# create or update the htpasswd file
|
||||
htpasswdFile = os.path.join(MalcolmPath, os.path.join('nginx', 'htpasswd'))
|
||||
htpasswdCmd = ['htpasswd',
|
||||
'-i',
|
||||
'-B',
|
||||
htpasswdFile,
|
||||
username]
|
||||
if not os.path.isfile(htpasswdFile):
|
||||
htpasswdCmd.insert(1, '-c')
|
||||
err, out = run_process(htpasswdCmd, stdin=password, stderr=True, debug=args.debug)
|
||||
if (err != 0):
|
||||
raise Exception(f'Unable to generate htpasswd file: {out}')
|
||||
|
||||
# if the admininstrator username has changed, remove the previous administrator username from htpasswd
|
||||
if (usernamePrevious is not None) and (usernamePrevious != username):
|
||||
htpasswdLines = list()
|
||||
with open(htpasswdFile, 'r') as f:
|
||||
htpasswdLines = f.readlines()
|
||||
with open(htpasswdFile, 'w') as f:
|
||||
for line in htpasswdLines:
|
||||
if not line.startswith(f"{usernamePrevious}:"):
|
||||
f.write(line)
|
||||
|
||||
# configure default LDAP stuff (they'll have to edit it by hand later)
|
||||
ldapConfFile = os.path.join(MalcolmPath, os.path.join('nginx', 'nginx_ldap.conf'))
|
||||
if not os.path.isfile(ldapConfFile):
|
||||
ldapDefaults = defaultdict(str)
|
||||
if os.path.isfile(os.path.join(MalcolmPath, '.ldap_config_defaults')):
|
||||
ldapDefaults = defaultdict(str)
|
||||
with open(os.path.join(MalcolmPath, '.ldap_config_defaults'), 'r') as f:
|
||||
for line in f:
|
||||
try:
|
||||
k, v = line.rstrip().split("=")
|
||||
ldapDefaults[k] = v.strip('"').strip("'")
|
||||
except:
|
||||
pass
|
||||
ldapProto = ldapDefaults.get("LDAP_PROTO", "ldap://")
|
||||
ldapHost = ldapDefaults.get("LDAP_HOST", "ds.example.com")
|
||||
ldapPort = ldapDefaults.get("LDAP_PORT", "3268")
|
||||
ldapType = ldapDefaults.get("LDAP_SERVER_TYPE", "winldap")
|
||||
if (ldapType == "openldap"):
|
||||
ldapUri = 'DC=example,DC=com?uid?sub?(objectClass=posixAccount)'
|
||||
ldapGroupAttr = "memberUid"
|
||||
ldapGroupAttrIsDN = "off"
|
||||
else:
|
||||
ldapUri = 'DC=example,DC=com?sAMAccountName?sub?(objectClass=person)'
|
||||
ldapGroupAttr = "member"
|
||||
ldapGroupAttrIsDN = "on"
|
||||
with open(ldapConfFile, 'w') as f:
|
||||
f.write('# This is a sample configuration for the ldap_server section of nginx.conf.\n')
|
||||
f.write('# Yours will vary depending on how your Active Directory/LDAP server is configured.\n')
|
||||
f.write('# See https://github.com/kvspb/nginx-auth-ldap#available-config-parameters for options.\n\n')
|
||||
f.write('ldap_server ad_server {\n')
|
||||
f.write(f' url "{ldapProto}{ldapHost}:{ldapPort}/{ldapUri}";\n\n')
|
||||
f.write(' binddn "bind_dn";\n')
|
||||
f.write(' binddn_passwd "bind_dn_password";\n\n')
|
||||
f.write(f' group_attribute {ldapGroupAttr};\n')
|
||||
f.write(f' group_attribute_is_dn {ldapGroupAttrIsDN};\n')
|
||||
f.write(' require group "CN=malcolm,OU=groups,DC=example,DC=com";\n')
|
||||
f.write(' require valid_user;\n')
|
||||
f.write(' satisfy all;\n')
|
||||
f.write('}\n\n')
|
||||
f.write('auth_ldap_cache_enabled on;\n')
|
||||
f.write('auth_ldap_cache_expiration_time 10000;\n')
|
||||
f.write('auth_ldap_cache_size 1000;\n')
|
||||
os.chmod(ldapConfFile, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
|
||||
|
||||
# populate htadmin config file
|
||||
with open(os.path.join(MalcolmPath, os.path.join('htadmin', 'config.ini')), 'w') as f:
|
||||
f.write('; HTAdmin config file.\n\n')
|
||||
f.write('[application]\n')
|
||||
f.write('; Change this to customize your title:\n')
|
||||
f.write('app_title = Malcolm User Management\n\n')
|
||||
f.write('; htpasswd file\n')
|
||||
f.write('secure_path = ./config/htpasswd\n')
|
||||
f.write('; metadata file\n')
|
||||
f.write('metadata_path = ./config/metadata\n\n')
|
||||
f.write('; administrator user/password (htpasswd -b -c -B ...)\n')
|
||||
f.write(f'admin_user = {username}\n\n')
|
||||
f.write('; username field quality checks\n')
|
||||
f.write(';\n')
|
||||
f.write('min_username_len = 4\n')
|
||||
f.write('max_username_len = 12\n\n')
|
||||
f.write('; Password field quality checks\n')
|
||||
f.write(';\n')
|
||||
f.write('min_password_len = 6\n')
|
||||
f.write('max_password_len = 20\n\n')
|
||||
|
||||
# touch the metadata file
|
||||
open(os.path.join(MalcolmPath, os.path.join('htadmin', 'metadata')), 'a').close()
|
||||
|
||||
# generate HTTPS self-signed certificates
|
||||
if YesOrNo('(Re)generate self-signed certificates for HTTPS access', default=True):
|
||||
with pushd(os.path.join(MalcolmPath, os.path.join('nginx', 'certs'))):
|
||||
# remove previous files
|
||||
for oldfile in glob.glob("*.pem"):
|
||||
os.remove(oldfile)
|
||||
|
||||
# generate dhparam -------------------------------
|
||||
err, out = run_process([opensslBin, 'dhparam', '-out', 'dhparam.pem', '2048'], stderr=True, debug=args.debug)
|
||||
if (err != 0):
|
||||
raise Exception(f'Unable to generate dhparam.pem file: {out}')
|
||||
|
||||
# generate key/cert -------------------------------
|
||||
err, out = run_process([opensslBin, 'req', '-subj', '/CN=localhost', '-x509', '-newkey', 'rsa:4096', '-nodes', '-keyout', 'key.pem', '-out', 'cert.pem', '-days', '3650'], stderr=True, debug=args.debug)
|
||||
if (err != 0):
|
||||
raise Exception(f'Unable to generate key.pem/cert.pem file(s): {out}')
|
||||
|
||||
# generate beats/logstash self-signed certificates
|
||||
logstashPath = os.path.join(MalcolmPath, os.path.join('logstash', 'certs'))
|
||||
filebeatPath = os.path.join(MalcolmPath, os.path.join('filebeat', 'certs'))
|
||||
if YesOrNo('(Re)generate self-signed certificates for a remote log forwarder', default=True):
|
||||
with pushd(logstashPath):
|
||||
|
||||
# make clean to clean previous files
|
||||
for pat in ['*.srl', '*.csr', '*.key', '*.crt', '*.pem']:
|
||||
for oldfile in glob.glob(pat):
|
||||
os.remove(oldfile)
|
||||
|
||||
# -----------------------------------------------
|
||||
# generate new ca/server/client certificates/keys
|
||||
# ca -------------------------------
|
||||
err, out = run_process([opensslBin, 'genrsa', '-out', 'ca.key', '2048'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate ca.key: {out}')
|
||||
|
||||
err, out = run_process([opensslBin, 'req', '-x509', '-new', '-nodes', '-key', 'ca.key', '-sha256', '-days', '9999', '-subj', '/C=US/ST=ID/O=sensor/OU=ca', '-out', 'ca.crt'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate ca.crt: {out}')
|
||||
|
||||
# server -------------------------------
|
||||
err, out = run_process([opensslBin, 'genrsa', '-out', 'server.key', '2048'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate server.key: {out}')
|
||||
|
||||
err, out = run_process([opensslBin, 'req', '-sha512', '-new', '-key', 'server.key', '-out', 'server.csr', '-config', 'server.conf'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate server.csr: {out}')
|
||||
|
||||
err, out = run_process([opensslBin, 'x509', '-days', '3650', '-req', '-sha512', '-in', 'server.csr', '-CAcreateserial', '-CA', 'ca.crt', '-CAkey', 'ca.key', '-out', 'server.crt', '-extensions', 'v3_req', '-extfile', 'server.conf'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate server.crt: {out}')
|
||||
|
||||
shutil.move("server.key", "server.key.pem")
|
||||
err, out = run_process([opensslBin, 'pkcs8', '-in', 'server.key.pem', '-topk8', '-nocrypt', '-out', 'server.key'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate server.key: {out}')
|
||||
|
||||
# client -------------------------------
|
||||
err, out = run_process([opensslBin, 'genrsa', '-out', 'client.key', '2048'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate client.key: {out}')
|
||||
|
||||
err, out = run_process([opensslBin, 'req', '-sha512', '-new', '-key', 'client.key', '-out', 'client.csr', '-config', 'client.conf'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate client.csr: {out}')
|
||||
|
||||
err, out = run_process([opensslBin, 'x509', '-days', '3650', '-req', '-sha512', '-in', 'client.csr', '-CAcreateserial', '-CA', 'ca.crt', '-CAkey', 'ca.key', '-out', 'client.crt', '-extensions', 'v3_req', '-extensions', 'usr_cert', '-extfile', 'client.conf'], stderr=True, debug=args.debug)
|
||||
if (err != 0): raise Exception(f'Unable to generate client.crt: {out}')
|
||||
# -----------------------------------------------
|
||||
|
||||
# mkdir filebeat/certs if it doesn't exist
|
||||
try:
|
||||
os.makedirs(filebeatPath)
|
||||
except OSError as exc:
|
||||
if (exc.errno == errno.EEXIST) and os.path.isdir(filebeatPath):
|
||||
pass
|
||||
else:
|
||||
raise
|
||||
|
||||
# remove previous files in filebeat/certs
|
||||
for oldfile in glob.glob(os.path.join(filebeatPath, "*")):
|
||||
os.remove(oldfile)
|
||||
|
||||
# copy the ca so logstasn and filebeat both have it
|
||||
shutil.copy2(os.path.join(logstashPath, "ca.crt"), filebeatPath)
|
||||
|
||||
# move the client certs for filebeat
|
||||
for f in ['client.key', 'client.crt']:
|
||||
shutil.move(os.path.join(logstashPath, f), filebeatPath)
|
||||
|
||||
# remove leftovers
|
||||
for pat in ['*.srl', '*.csr', '*.pem']:
|
||||
for oldfile in glob.glob(pat):
|
||||
os.remove(oldfile)
|
||||
|
||||
# create and populate keystore for remote
|
||||
if YesOrNo('Store username/password for forwarding Logstash events to a secondary, external Elasticsearch instance', default=False):
|
||||
|
||||
# prompt username and password
|
||||
esPassword = None
|
||||
esPasswordConfirm = None
|
||||
esUsername = AskForString("External Elasticsearch username")
|
||||
|
||||
while True:
|
||||
esPassword = AskForPassword(f"{esUsername} password: ")
|
||||
esPasswordConfirm = AskForPassword(f"{esUsername} password (again): ")
|
||||
if (esPassword == esPasswordConfirm):
|
||||
break
|
||||
eprint("Passwords do not match")
|
||||
|
||||
# create logstash keystore file, don't complain if it already exists, and set the keystore items
|
||||
keystore_op('logstash', False, 'create', stdin='N')
|
||||
keystore_op('logstash', False, 'remove', 'ES_EXTERNAL_USER', '--force')
|
||||
keystore_op('logstash', False, 'add', 'ES_EXTERNAL_USER', '--stdin', '--force', stdin=esUsername)
|
||||
keystore_op('logstash', False, 'remove', 'ES_EXTERNAL_PASSWORD', '--force')
|
||||
keystore_op('logstash', False, 'add', 'ES_EXTERNAL_PASSWORD', '--stdin', '--force', stdin=esPassword)
|
||||
success, results = keystore_op('logstash', False, 'list')
|
||||
results = [x.upper() for x in results if x and (not x.upper().startswith('WARNING')) and (not x.upper().startswith('KEYSTORE')) and (not x.upper().startswith('USING BUNDLED JDK'))]
|
||||
if success and ('ES_EXTERNAL_USER' in results) and ('ES_EXTERNAL_PASSWORD' in results):
|
||||
eprint(f"External Elasticsearch instance variables stored: {', '.join(results)}")
|
||||
else:
|
||||
eprint("Failed to store external Elasticsearch instance variables:\n")
|
||||
eprint("\n".join(results))
|
||||
|
||||
# Open Distro for Elasticsearch authenticate sender account credentials
|
||||
# https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/monitors/#authenticate-sender-account
|
||||
if YesOrNo('Store username/password for email alert sender account (see https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/monitors/#authenticate-sender-account)', default=False):
|
||||
|
||||
# prompt username and password
|
||||
emailPassword = None
|
||||
emailPasswordConfirm = None
|
||||
emailSender = AskForString("Open Distro alerting email sender name")
|
||||
emailUsername = AskForString("Email account username")
|
||||
|
||||
while True:
|
||||
emailPassword = AskForPassword(f"{emailUsername} password: ")
|
||||
emailPasswordConfirm = AskForPassword(f"{emailUsername} password (again): ")
|
||||
if (emailPassword == emailPasswordConfirm):
|
||||
break
|
||||
eprint("Passwords do not match")
|
||||
|
||||
# create elasticsearch keystore file, don't complain if it already exists, and set the keystore items
|
||||
usernameKey = f'opendistro.alerting.destination.email.{emailSender}.username'
|
||||
passwordKey = f'opendistro.alerting.destination.email.{emailSender}.password'
|
||||
|
||||
keystore_op('elasticsearch', True, 'create', stdin='N')
|
||||
keystore_op('elasticsearch', True, 'remove', usernameKey)
|
||||
keystore_op('elasticsearch', True, 'add', usernameKey, '--stdin', stdin=emailUsername)
|
||||
keystore_op('elasticsearch', True, 'remove', passwordKey)
|
||||
keystore_op('elasticsearch', True, 'add', passwordKey, '--stdin', stdin=emailPassword)
|
||||
success, results = keystore_op('elasticsearch', True, 'list')
|
||||
results = [x for x in results if x and (not x.upper().startswith('WARNING')) and (not x.upper().startswith('KEYSTORE'))]
|
||||
if success and (usernameKey in results) and (passwordKey in results):
|
||||
eprint(f"Email alert sender account variables stored: {', '.join(results)}")
|
||||
else:
|
||||
eprint("Failed to store email alert sender account variables:\n")
|
||||
eprint("\n".join(results))
|
||||
|
||||
###################################################################################################
|
||||
# main
|
||||
def main():
|
||||
global args
|
||||
global dockerBin
|
||||
global dockerComposeBin
|
||||
global opensslBin
|
||||
|
||||
# extract arguments from the command line
|
||||
# print (sys.argv[1:]);
|
||||
parser = argparse.ArgumentParser(description='Malcolm control script', add_help=False, usage=f'{ScriptName} <arguments>')
|
||||
parser.add_argument('-v', '--verbose', dest='debug', type=str2bool, nargs='?', const=True, default=False, help="Verbose output")
|
||||
parser.add_argument('-f', '--file', required=False, dest='composeFile', metavar='<STR>', type=str, default='docker-compose.yml', help='docker-compose YML file')
|
||||
parser.add_argument('-l', '--logs', dest='cmdLogs', type=str2bool, nargs='?', const=True, default=False, help="Tail Malcolm logs")
|
||||
parser.add_argument('--start', dest='cmdStart', type=str2bool, nargs='?', const=True, default=False, help="Start Malcolm")
|
||||
parser.add_argument('--restart', dest='cmdRestart', type=str2bool, nargs='?', const=True, default=False, help="Stop and restart Malcolm")
|
||||
parser.add_argument('--stop', dest='cmdStop', type=str2bool, nargs='?', const=True, default=False, help="Stop Malcolm")
|
||||
parser.add_argument('--wipe', dest='cmdWipe', type=str2bool, nargs='?', const=True, default=False, help="Stop Malcolm and delete all data")
|
||||
parser.add_argument('--auth', dest='cmdAuthSetup', type=str2bool, nargs='?', const=True, default=False, help="Configure Malcolm authentication")
|
||||
parser.add_argument('--status', dest='cmdStatus', type=str2bool, nargs='?', const=True, default=False, help="Display status of Malcolm components")
|
||||
|
||||
try:
|
||||
parser.error = parser.exit
|
||||
args = parser.parse_args()
|
||||
except SystemExit:
|
||||
parser.print_help()
|
||||
exit(2)
|
||||
|
||||
if args.debug:
|
||||
eprint(os.path.join(ScriptPath, ScriptName))
|
||||
eprint(f"Arguments: {sys.argv[1:]}")
|
||||
eprint(f"Arguments: {args}")
|
||||
eprint("Malcolm path:", MalcolmPath)
|
||||
else:
|
||||
sys.tracebacklimit = 0
|
||||
|
||||
with pushd(MalcolmPath):
|
||||
|
||||
# don't run this as root
|
||||
if (pyPlatform != PLATFORM_WINDOWS) and ((os.getuid() == 0) or (os.geteuid() == 0) or (getpass.getuser() == 'root')):
|
||||
raise Exception(f'{ScriptName} should not be run as root')
|
||||
|
||||
# create local temporary directory for docker-compose because we may have noexec on /tmp
|
||||
try:
|
||||
os.makedirs(MalcolmTmpPath)
|
||||
except OSError as exc:
|
||||
if (exc.errno == errno.EEXIST) and os.path.isdir(MalcolmTmpPath):
|
||||
pass
|
||||
else:
|
||||
raise
|
||||
|
||||
# docker-compose use local temporary path
|
||||
osEnv = os.environ.copy()
|
||||
osEnv['TMPDIR'] = MalcolmTmpPath
|
||||
|
||||
# make sure docker/docker-compose is available
|
||||
dockerBin = 'docker.exe' if ((pyPlatform == PLATFORM_WINDOWS) and Which('docker.exe')) else 'docker'
|
||||
dockerComposeBin = 'docker-compose.exe' if ((pyPlatform == PLATFORM_WINDOWS) and Which('docker-compose.exe')) else 'docker-compose'
|
||||
err, out = run_process([dockerBin, 'info'], debug=args.debug)
|
||||
if (err != 0):
|
||||
raise Exception(f'{ScriptName} requires docker, please run install.py')
|
||||
err, out = run_process([dockerComposeBin, '-f', args.composeFile, 'version'], env=osEnv, debug=args.debug)
|
||||
if (err != 0):
|
||||
raise Exception(f'{ScriptName} requires docker-compose, please run install.py')
|
||||
|
||||
# identify openssl binary
|
||||
opensslBin = 'openssl.exe' if ((pyPlatform == PLATFORM_WINDOWS) and Which('openssl.exe')) else 'openssl'
|
||||
|
||||
# if executed via a symlink, figure out what was intended via the symlink name
|
||||
if os.path.islink(os.path.join(ScriptPath, ScriptName)):
|
||||
if (ScriptName == "logs"):
|
||||
args.cmdLogs = True
|
||||
elif (ScriptName == "status"):
|
||||
args.cmdStatus = True
|
||||
elif (ScriptName == "start"):
|
||||
args.cmdStart = True
|
||||
elif (ScriptName == "restart"):
|
||||
args.cmdRestart = True
|
||||
elif (ScriptName == "stop"):
|
||||
args.cmdStop = True
|
||||
elif (ScriptName == "wipe"):
|
||||
args.cmdWipe = True
|
||||
elif (ScriptName.startswith("auth")):
|
||||
args.cmdAuthSetup = True
|
||||
|
||||
# stop Malcolm (and wipe data if requestsed)
|
||||
if args.cmdRestart or args.cmdStop or args.cmdWipe:
|
||||
stop(wipe=args.cmdWipe)
|
||||
|
||||
# configure Malcolm authentication
|
||||
if args.cmdAuthSetup:
|
||||
authSetup()
|
||||
|
||||
# start Malcolm
|
||||
if args.cmdStart or args.cmdRestart:
|
||||
start()
|
||||
|
||||
# tail Malcolm logs
|
||||
if args.cmdStart or args.cmdRestart or args.cmdLogs:
|
||||
logs()
|
||||
|
||||
# display Malcolm status
|
||||
if args.cmdStatus:
|
||||
status()
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
if coloramaImported:
|
||||
print(Style.RESET_ALL)
|
||||
|
||||
1333
Vagrant/resources/malcolm/scripts/install.py
Executable file
1333
Vagrant/resources/malcolm/scripts/install.py
Executable file
File diff suppressed because it is too large
Load Diff
1
Vagrant/resources/malcolm/scripts/logs
Symbolic link
1
Vagrant/resources/malcolm/scripts/logs
Symbolic link
@@ -0,0 +1 @@
|
||||
./control.py
|
||||
158
Vagrant/resources/malcolm/scripts/malcolm_appliance_packager.sh
Executable file
158
Vagrant/resources/malcolm/scripts/malcolm_appliance_packager.sh
Executable file
@@ -0,0 +1,158 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
if [ -z "$BASH_VERSION" ]; then
|
||||
echo "Wrong interpreter, please run \"$0\" with bash"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set -e
|
||||
|
||||
DESTDIR="$(mktemp -d -t malcolm-XXXXXX)"
|
||||
VERBOSE=""
|
||||
|
||||
function cleanup {
|
||||
if ! rm -rf "$DESTDIR"; then
|
||||
echo "Failed to remove temporary directory '$DESTDIR'"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# force-navigate to Malcolm base directory (parent of scripts/ directory)
|
||||
RUN_PATH="$(pwd)"
|
||||
[[ "$(uname -s)" = 'Darwin' ]] && REALPATH=grealpath || REALPATH=realpath
|
||||
[[ "$(uname -s)" = 'Darwin' ]] && DIRNAME=gdirname || DIRNAME=dirname
|
||||
if ! (type "$REALPATH" && type "$DIRNAME") > /dev/null; then
|
||||
echo "$(basename "${BASH_SOURCE[0]}") requires $REALPATH and $DIRNAME"
|
||||
exit 1
|
||||
fi
|
||||
SCRIPT_PATH="$($DIRNAME $($REALPATH -e "${BASH_SOURCE[0]}"))"
|
||||
pushd "$SCRIPT_PATH/.." >/dev/null 2>&1
|
||||
|
||||
CURRENT_REV_SHA="$(git rev-parse --short --verify HEAD)"
|
||||
if [ -z "$CURRENT_REV_SHA" ]; then
|
||||
CURRENT_REV_TAG="$(date +%Y.%m.%d_%H:%M:%S)"
|
||||
else
|
||||
CURRENT_REV_DATE="$(git log -1 --format="%at" | xargs -I{} date -d @{} +%Y%m%d_%H%M%S)"
|
||||
if [ -z "$CURRENT_REV_DATE" ]; then
|
||||
CURRENT_REV_TAG="$(date +%Y.%m.%d_%H:%M:%S)"
|
||||
fi
|
||||
CURRENT_REV_TAG="${CURRENT_REV_DATE}_${CURRENT_REV_SHA}"
|
||||
fi
|
||||
|
||||
DESTDIR="/tmp/malcolm_${CURRENT_REV_TAG}"
|
||||
if [ -d "$DESTDIR" ]; then
|
||||
unset CONFIRMATION
|
||||
echo ""
|
||||
read -p "Temporary directory \"$DESTDIR\" exists, delete before proceeding? [y/N]? " CONFIRMATION
|
||||
CONFIRMATION=${CONFIRMATION:-N}
|
||||
if [[ $CONFIRMATION =~ ^[Yy]$ ]]; then
|
||||
rm -rf "$DESTDIR"
|
||||
else
|
||||
echo "Aborting"
|
||||
popd >/dev/null 2>&1
|
||||
popd >/dev/null 2>&1
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if mkdir "$DESTDIR"; then
|
||||
# ensure that if we "grabbed a lock", we release it (works for clean exit, SIGTERM, and SIGINT/Ctrl-C)
|
||||
trap "cleanup" EXIT
|
||||
|
||||
mkdir $VERBOSE -p "$DESTDIR/nginx/certs/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/nginx/ca-trust/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/htadmin/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/logstash/certs/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/filebeat/certs/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/elasticsearch/nodes/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/elasticsearch-backup/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/moloch-raw/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/moloch-logs/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/pcap/upload/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/pcap/processed/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/yara/rules/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/zeek-logs/current/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/zeek-logs/upload/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/zeek-logs/processed/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/zeek-logs/extract_files/"
|
||||
mkdir $VERBOSE -p "$DESTDIR/scripts/"
|
||||
cp $VERBOSE ./docker-compose-standalone.yml "$DESTDIR/docker-compose.yml"
|
||||
cp $VERBOSE ./auth.env "$DESTDIR/"
|
||||
cp $VERBOSE ./cidr-map.txt "$DESTDIR/"
|
||||
cp $VERBOSE ./host-map.txt "$DESTDIR/"
|
||||
cp $VERBOSE ./net-map.json "$DESTDIR/"
|
||||
cp $VERBOSE ./index-management-policy.json "$DESTDIR/"
|
||||
cp $VERBOSE ./scripts/install.py "$DESTDIR/scripts/"
|
||||
cp $VERBOSE ./scripts/control.py "$DESTDIR/scripts/"
|
||||
cp $VERBOSE ./scripts/malcolm_common.py "$DESTDIR/scripts/"
|
||||
cp $VERBOSE ./README.md "$DESTDIR/"
|
||||
cp $VERBOSE ./logstash/certs/*.conf "$DESTDIR/logstash/certs/"
|
||||
pushd "$DESTDIR" >/dev/null 2>&1
|
||||
pushd "./scripts" >/dev/null 2>&1
|
||||
ln -s ./control.py auth_setup
|
||||
ln -s ./control.py logs
|
||||
ln -s ./control.py restart
|
||||
ln -s ./control.py start
|
||||
ln -s ./control.py status
|
||||
ln -s ./control.py stop
|
||||
ln -s ./control.py wipe
|
||||
popd >/dev/null 2>&1
|
||||
echo "You must set an administrator username and password for Malcolm, and self-signed X.509 certificates will be generated"
|
||||
./scripts/auth_setup
|
||||
rm -rf logstash/certs/ca.key
|
||||
pushd .. >/dev/null 2>&1
|
||||
DESTNAME="$RUN_PATH/$(basename $DESTDIR).tar.gz"
|
||||
README="$RUN_PATH/$(basename $DESTDIR).README.txt"
|
||||
README_HTML="$RUN_PATH/$(basename $DESTDIR).README.html"
|
||||
docker run --rm --entrypoint /bin/bash "$(grep -E 'image: *malcolmnetsec/arkime' "$DESTDIR/docker-compose.yml" | awk '{print $2}')" -c "cat /data/moloch/doc/README.html" > "$README_HTML" || true
|
||||
cp $VERBOSE "$SCRIPT_PATH/install.py" "$RUN_PATH/"
|
||||
cp $VERBOSE "$SCRIPT_PATH/malcolm_common.py" "$RUN_PATH/"
|
||||
tar -czf $VERBOSE "$DESTNAME" "./$(basename $DESTDIR)/"
|
||||
echo "Packaged Malcolm to \"$DESTNAME\""
|
||||
echo ""
|
||||
|
||||
unset CONFIRMATION
|
||||
echo ""
|
||||
read -p "Do you need to package docker images also [y/N]? " CONFIRMATION
|
||||
CONFIRMATION=${CONFIRMATION:-N}
|
||||
if [[ $CONFIRMATION =~ ^[Yy]$ ]]; then
|
||||
echo "This might take a few minutes..."
|
||||
DESTNAMEIMAGES="$RUN_PATH/$(basename $DESTDIR)_images.tar.gz"
|
||||
IMAGES=( $(grep image: $DESTDIR/docker-compose.yml | awk '{print $2}') )
|
||||
docker save "${IMAGES[@]}" | gzip > "$DESTNAMEIMAGES"
|
||||
echo "Packaged Malcolm docker images to \"$DESTNAMEIMAGES\""
|
||||
echo ""
|
||||
fi
|
||||
echo ""
|
||||
echo "To install Malcolm:" | tee -a "$README"
|
||||
echo " 1. Run install.py" | tee -a "$README"
|
||||
echo " 2. Follow the prompts" | tee -a "$README"
|
||||
echo "" | tee -a "$README"
|
||||
echo "To start, stop, restart, etc. Malcolm:" | tee -a "$README"
|
||||
echo " Use the control scripts in the \"scripts/\" directory:" | tee -a "$README"
|
||||
echo " - start (start Malcolm)" | tee -a "$README"
|
||||
echo " - stop (stop Malcolm)" | tee -a "$README"
|
||||
echo " - restart (restart Malcolm)" | tee -a "$README"
|
||||
echo " - logs (monitor Malcolm logs)" | tee -a "$README"
|
||||
echo " - wipe (stop Malcolm and clear its database)" | tee -a "$README"
|
||||
echo " - auth_setup (change authentication-related settings)" | tee -a "$README"
|
||||
echo "" | tee -a "$README"
|
||||
echo "A minute or so after starting Malcolm, the following services will be accessible:" | tee -a "$README"
|
||||
echo " - Arkime: https://localhost/" | tee -a "$README"
|
||||
echo " - Kibana: https://localhost/kibana/" | tee -a "$README"
|
||||
echo " - PCAP upload (web): https://localhost/upload/" | tee -a "$README"
|
||||
echo " - PCAP upload (sftp): sftp://USERNAME@127.0.0.1:8022/files/" | tee -a "$README"
|
||||
echo " - Host and subnet name mapping editor: https://localhost/name-map-ui/" | tee -a "$README"
|
||||
echo " - Account management: https://localhost:488/" | tee -a "$README"
|
||||
popd >/dev/null 2>&1
|
||||
popd >/dev/null 2>&1
|
||||
popd >/dev/null 2>&1
|
||||
popd >/dev/null 2>&1
|
||||
else
|
||||
echo "Unable to create temporary directory \"$DESTDIR\""
|
||||
popd >/dev/null 2>&1
|
||||
popd >/dev/null 2>&1
|
||||
exit 1
|
||||
fi
|
||||
319
Vagrant/resources/malcolm/scripts/malcolm_common.py
Normal file
319
Vagrant/resources/malcolm/scripts/malcolm_common.py
Normal file
@@ -0,0 +1,319 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
import contextlib
|
||||
import getpass
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
from collections import defaultdict
|
||||
try:
|
||||
from pwd import getpwuid
|
||||
except ImportError:
|
||||
getpwuid = None
|
||||
from subprocess import (PIPE, STDOUT, Popen, CalledProcessError)
|
||||
|
||||
###################################################################################################
|
||||
ScriptPath = os.path.dirname(os.path.realpath(__file__))
|
||||
MalcolmPath = os.path.abspath(os.path.join(ScriptPath, os.pardir))
|
||||
MalcolmTmpPath = os.path.join(MalcolmPath, '.tmp')
|
||||
|
||||
###################################################################################################
|
||||
|
||||
# attempt to import requests, will cover failure later
|
||||
try:
|
||||
import requests
|
||||
RequestsImported = True
|
||||
except ImportError:
|
||||
RequestsImported = False
|
||||
|
||||
###################################################################################################
|
||||
PLATFORM_WINDOWS = "Windows"
|
||||
PLATFORM_MAC = "Darwin"
|
||||
PLATFORM_LINUX = "Linux"
|
||||
PLATFORM_LINUX_CENTOS = 'centos'
|
||||
PLATFORM_LINUX_DEBIAN = 'debian'
|
||||
PLATFORM_LINUX_FEDORA = 'fedora'
|
||||
PLATFORM_LINUX_UBUNTU = 'ubuntu'
|
||||
|
||||
# URLS for figuring things out if something goes wrong
|
||||
DOCKER_INSTALL_URLS = defaultdict(lambda: 'https://docs.docker.com/install/')
|
||||
DOCKER_INSTALL_URLS[PLATFORM_WINDOWS] = ['https://stefanscherer.github.io/how-to-install-docker-the-chocolatey-way/',
|
||||
'https://docs.docker.com/docker-for-windows/install/']
|
||||
DOCKER_INSTALL_URLS[PLATFORM_LINUX_UBUNTU] = 'https://docs.docker.com/install/linux/docker-ce/ubuntu/'
|
||||
DOCKER_INSTALL_URLS[PLATFORM_LINUX_DEBIAN] = 'https://docs.docker.com/install/linux/docker-ce/debian/'
|
||||
DOCKER_INSTALL_URLS[PLATFORM_LINUX_CENTOS] = 'https://docs.docker.com/install/linux/docker-ce/centos/'
|
||||
DOCKER_INSTALL_URLS[PLATFORM_LINUX_FEDORA] = 'https://docs.docker.com/install/linux/docker-ce/fedora/'
|
||||
DOCKER_INSTALL_URLS[PLATFORM_MAC] = ['https://www.code2bits.com/how-to-install-docker-on-macos-using-homebrew/',
|
||||
'https://docs.docker.com/docker-for-mac/install/']
|
||||
DOCKER_COMPOSE_INSTALL_URLS = defaultdict(lambda: 'https://docs.docker.com/compose/install/')
|
||||
HOMEBREW_INSTALL_URLS = defaultdict(lambda: 'https://brew.sh/')
|
||||
|
||||
###################################################################################################
|
||||
# chdir to directory as context manager, returning automatically
|
||||
@contextlib.contextmanager
|
||||
def pushd(directory):
|
||||
prevDir = os.getcwd()
|
||||
os.chdir(directory)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
os.chdir(prevDir)
|
||||
|
||||
###################################################################################################
|
||||
# print to stderr
|
||||
def eprint(*args, **kwargs):
|
||||
print(*args, file=sys.stderr, **kwargs)
|
||||
|
||||
###################################################################################################
|
||||
def EscapeAnsi(line):
|
||||
ansiEscape = re.compile(r'(?:\x1B[@-_]|[\x80-\x9F])[0-?]*[ -/]*[@-~]')
|
||||
return ansiEscape.sub('', line)
|
||||
|
||||
###################################################################################################
|
||||
# get interactive user response to Y/N question
|
||||
def YesOrNo(question, default=None, forceInteraction=False, acceptDefault=False):
|
||||
|
||||
if default == True:
|
||||
questionStr = f"\n{question} (Y/n): "
|
||||
elif default == False:
|
||||
questionStr = f"\n{question} (y/N): "
|
||||
else:
|
||||
questionStr = f"\n{question} (y/n): "
|
||||
|
||||
if acceptDefault and (default is not None) and (not forceInteraction):
|
||||
reply = ''
|
||||
else:
|
||||
while True:
|
||||
reply = str(input(questionStr)).lower().strip()
|
||||
if (len(reply) > 0) or (default is not None):
|
||||
break
|
||||
|
||||
if (len(reply) == 0):
|
||||
reply = 'y' if default else 'n'
|
||||
|
||||
if reply[0] == 'y':
|
||||
return True
|
||||
elif reply[0] == 'n':
|
||||
return False
|
||||
else:
|
||||
return YesOrNo(question, default=default)
|
||||
|
||||
###################################################################################################
|
||||
# get interactive user response
|
||||
def AskForString(question, default=None, forceInteraction=False, acceptDefault=False):
|
||||
|
||||
if acceptDefault and (default is not None) and (not forceInteraction):
|
||||
reply = default
|
||||
else:
|
||||
reply = str(input(f'\n{question}: ')).strip()
|
||||
|
||||
return reply
|
||||
|
||||
###################################################################################################
|
||||
# get interactive password (without echoing)
|
||||
def AskForPassword(prompt):
|
||||
reply = getpass.getpass(prompt=prompt)
|
||||
return reply
|
||||
|
||||
###################################################################################################
|
||||
# convenient boolean argument parsing
|
||||
def str2bool(v):
|
||||
if v.lower() in ('yes', 'true', 't', 'y', '1'):
|
||||
return True
|
||||
elif v.lower() in ('no', 'false', 'f', 'n', '0'):
|
||||
return False
|
||||
else:
|
||||
raise ValueError('Boolean value expected')
|
||||
|
||||
###################################################################################################
|
||||
# determine if a program/script exists and is executable in the system path
|
||||
def Which(cmd, debug=False):
|
||||
result = any(os.access(os.path.join(path, cmd), os.X_OK) for path in os.environ["PATH"].split(os.pathsep))
|
||||
if debug:
|
||||
eprint(f"Which {cmd} returned {result}")
|
||||
return result
|
||||
|
||||
###################################################################################################
|
||||
# nice human-readable file sizes
|
||||
def SizeHumanFormat(num, suffix='B'):
|
||||
for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:
|
||||
if abs(num) < 1024.0:
|
||||
return f"{num:3.1f}{unit}{suffix}"
|
||||
num /= 1024.0
|
||||
return f"{num:.1f}{'Yi'}{suffix}"
|
||||
|
||||
###################################################################################################
|
||||
# is this string valid json? if so, load and return it
|
||||
def LoadStrIfJson(jsonStr):
|
||||
try:
|
||||
return json.loads(jsonStr)
|
||||
except ValueError as e:
|
||||
return None
|
||||
|
||||
###################################################################################################
|
||||
# run command with arguments and return its exit code, stdout, and stderr
|
||||
def check_output_input(*popenargs, **kwargs):
|
||||
|
||||
if 'stdout' in kwargs:
|
||||
raise ValueError('stdout argument not allowed, it will be overridden')
|
||||
|
||||
if 'stderr' in kwargs:
|
||||
raise ValueError('stderr argument not allowed, it will be overridden')
|
||||
|
||||
if 'input' in kwargs and kwargs['input']:
|
||||
if 'stdin' in kwargs:
|
||||
raise ValueError('stdin and input arguments may not both be used')
|
||||
inputdata = kwargs['input']
|
||||
kwargs['stdin'] = PIPE
|
||||
else:
|
||||
inputdata = None
|
||||
kwargs.pop('input', None)
|
||||
|
||||
process = Popen(*popenargs, stdout=PIPE, stderr=PIPE, **kwargs)
|
||||
try:
|
||||
output, errput = process.communicate(inputdata)
|
||||
except:
|
||||
process.kill()
|
||||
process.wait()
|
||||
raise
|
||||
|
||||
retcode = process.poll()
|
||||
|
||||
return retcode, output, errput
|
||||
|
||||
###################################################################################################
|
||||
# run command with arguments and return its exit code, stdout, and stderr
|
||||
def run_process(command, stdout=True, stderr=True, stdin=None, retry=0, retrySleepSec=5, cwd=None, env=None, debug=False):
|
||||
|
||||
retcode = -1
|
||||
output = []
|
||||
|
||||
try:
|
||||
# run the command
|
||||
retcode, cmdout, cmderr = check_output_input(command, input=stdin.encode() if stdin else stdin, cwd=cwd, env=env)
|
||||
|
||||
# split the output on newlines to return a list
|
||||
if stderr and (len(cmderr) > 0): output.extend(cmderr.decode(sys.getdefaultencoding()).split('\n'))
|
||||
if stdout and (len(cmdout) > 0): output.extend(cmdout.decode(sys.getdefaultencoding()).split('\n'))
|
||||
|
||||
except (FileNotFoundError, OSError, IOError) as e:
|
||||
if stderr:
|
||||
output.append(f"Command {command} not found or unable to execute")
|
||||
|
||||
if debug:
|
||||
eprint(f"{command}({stdin[:80] + bool(stdin[80:]) * '...' if stdin else ''}) returned {retcode}: {output}")
|
||||
|
||||
if (retcode != 0) and retry and (retry > 0):
|
||||
# sleep then retry
|
||||
time.sleep(retrySleepSec)
|
||||
return run_process(command, stdout, stderr, stdin, retry-1, retrySleepSec, cwd, env, debug)
|
||||
else:
|
||||
return retcode, output
|
||||
|
||||
###################################################################################################
|
||||
# make sure we can import requests properly and take care of it automatically if possible
|
||||
def ImportRequests(debug=False):
|
||||
global RequestsImported
|
||||
|
||||
if not RequestsImported:
|
||||
# see if we can help out by installing the requests module
|
||||
|
||||
pyPlatform = platform.system()
|
||||
pyExec = sys.executable
|
||||
pipCmd = 'pip3'
|
||||
if not Which(pipCmd, debug=debug): pipCmd = 'pip'
|
||||
|
||||
eprint(f'The requests module is required under Python {platform.python_version()} ({pyExec})')
|
||||
|
||||
if Which(pipCmd, debug=debug):
|
||||
if YesOrNo(f'Importing the requests module failed. Attempt to install via {pipCmd}?'):
|
||||
installCmd = None
|
||||
|
||||
if (pyPlatform == PLATFORM_LINUX) or (pyPlatform == PLATFORM_MAC):
|
||||
# for linux/mac, we're going to try to figure out if this python is owned by root or the script user
|
||||
if (getpass.getuser() == getpwuid(os.stat(pyExec).st_uid).pw_name):
|
||||
# we're running a user-owned python, regular pip should work
|
||||
installCmd = [pipCmd, 'install', 'requests']
|
||||
else:
|
||||
# python is owned by system, so make sure to pass the --user flag
|
||||
installCmd = [pipCmd, 'install', '--user', 'requests']
|
||||
else:
|
||||
# on windows (or whatever other platform this is) I don't know any other way other than pip
|
||||
installCmd = [pipCmd, 'install', 'requests']
|
||||
|
||||
err, out = run_process(installCmd, debug=debug)
|
||||
if err == 0:
|
||||
eprint("Installation of requests module apparently succeeded")
|
||||
try:
|
||||
import requests
|
||||
RequestsImported = True
|
||||
except ImportError as e:
|
||||
eprint(f"Importing the requests module still failed: {e}")
|
||||
else:
|
||||
eprint(f"Installation of requests module failed: {out}")
|
||||
|
||||
if not RequestsImported:
|
||||
eprint("System-wide installation varies by platform and Python configuration. Please consult platform-specific documentation for installing Python modules.")
|
||||
if (platform.system() == PLATFORM_MAC):
|
||||
eprint('You *may* be able to install pip and requests manually via: sudo sh -c "easy_install pip && pip install requests"')
|
||||
elif (pyPlatform == PLATFORM_LINUX):
|
||||
if Which('apt-get', debug=debug):
|
||||
eprint("You *may* be able to install requests manually via: sudo apt-get install python3-requests")
|
||||
elif Which('apt', debug=debug):
|
||||
eprint("You *may* be able to install requests manually via: sudo apt install python3-requests")
|
||||
elif Which('dnf', debug=debug):
|
||||
eprint("You *may* be able to install requests manually via: sudo dnf install python3-requests")
|
||||
elif Which('yum', debug=debug):
|
||||
eprint('You *may* be able to install pip and requests manually via: sudo sh -c "yum install python3-pip && python3 -m pip install requests"')
|
||||
|
||||
return RequestsImported
|
||||
|
||||
###################################################################################################
|
||||
# do the required auth files for Malcolm exist?
|
||||
def MalcolmAuthFilesExist():
|
||||
return os.path.isfile(os.path.join(MalcolmPath, os.path.join('nginx', 'htpasswd'))) and \
|
||||
os.path.isfile(os.path.join(MalcolmPath, os.path.join('nginx', 'nginx_ldap.conf'))) and \
|
||||
os.path.isfile(os.path.join(MalcolmPath, os.path.join('nginx', os.path.join('certs', 'cert.pem')))) and \
|
||||
os.path.isfile(os.path.join(MalcolmPath, os.path.join('nginx', os.path.join('certs', 'key.pem')))) and \
|
||||
os.path.isfile(os.path.join(MalcolmPath, os.path.join('htadmin', 'config.ini'))) and \
|
||||
os.path.isfile(os.path.join(MalcolmPath, 'auth.env'))
|
||||
|
||||
###################################################################################################
|
||||
# download to file
|
||||
def DownloadToFile(url, local_filename, debug=False):
|
||||
r = requests.get(url, stream=True, allow_redirects=True)
|
||||
with open(local_filename, 'wb') as f:
|
||||
for chunk in r.iter_content(chunk_size=1024):
|
||||
if chunk: f.write(chunk)
|
||||
fExists = os.path.isfile(local_filename)
|
||||
fSize = os.path.getsize(local_filename)
|
||||
if debug:
|
||||
eprint(f"Download of {url} to {local_filename} {'succeeded' if fExists else 'failed'} ({SizeHumanFormat(fSize)})")
|
||||
return fExists and (fSize > 0)
|
||||
|
||||
###################################################################################################
|
||||
# recursively remove empty subfolders
|
||||
def RemoveEmptyFolders(path, removeRoot=True):
|
||||
if not os.path.isdir(path):
|
||||
return
|
||||
|
||||
files = os.listdir(path)
|
||||
if len(files):
|
||||
for f in files:
|
||||
fullpath = os.path.join(path, f)
|
||||
if os.path.isdir(fullpath):
|
||||
RemoveEmptyFolders(fullpath)
|
||||
|
||||
files = os.listdir(path)
|
||||
if len(files) == 0 and removeRoot:
|
||||
try:
|
||||
os.rmdir(path)
|
||||
except:
|
||||
pass
|
||||
121
Vagrant/resources/malcolm/scripts/package_zeek_logs.sh
Executable file
121
Vagrant/resources/malcolm/scripts/package_zeek_logs.sh
Executable file
@@ -0,0 +1,121 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
|
||||
|
||||
# package up Zeek logs in a format more suitable for upload to Malcolm
|
||||
#
|
||||
# directory containing Zeek logs is a parent directory of directories/files named like smb_mapping.04/00/00-05/00/00.log.gz
|
||||
#
|
||||
|
||||
set -e
|
||||
set -u
|
||||
set -o pipefail
|
||||
|
||||
ENCODING="utf-8"
|
||||
|
||||
# options
|
||||
# -v (verbose)
|
||||
# -d dir (base directory containing logs, e.g., the parent directory of smb_mapping.04/00/00-05/00/00.log.gz )
|
||||
|
||||
# parse command-line options
|
||||
VERBOSE_FLAG=""
|
||||
LOG_BASE_DIR=$(pwd)
|
||||
while getopts 'vd:' OPTION; do
|
||||
case "$OPTION" in
|
||||
v)
|
||||
VERBOSE_FLAG="-v"
|
||||
;;
|
||||
|
||||
d)
|
||||
LOG_BASE_DIR="$OPTARG"
|
||||
;;
|
||||
|
||||
?)
|
||||
echo "script usage: $(basename $0) [-v] [-d directory]" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
shift "$(($OPTIND -1))"
|
||||
|
||||
# fsize - display byte sizes human readable
|
||||
function fsize () {
|
||||
echo "$1" | awk 'function human(x) {
|
||||
s=" B KiB MiB GiB TiB EiB PiB YiB ZiB"
|
||||
while (x>=1024 && length(s)>1)
|
||||
{x/=1024; s=substr(s,5)}
|
||||
s=substr(s,1,4)
|
||||
xf=(s==" B ")?"%5d ":"%0.2f"
|
||||
return sprintf( xf"%s", x, s)
|
||||
}
|
||||
{gsub(/^[0-9]+/, human($1)); print}'
|
||||
}
|
||||
|
||||
function fdir () {
|
||||
[[ -f "$1" ]] && echo "$(dirname "$1")" || echo "$1"
|
||||
}
|
||||
|
||||
# create a temporary directory to store our results in (make sure /tmp is big enough to extract all of these logs into!)
|
||||
WORKDIR="$(mktemp -d -t malcolm-zeek-XXXXXX)"
|
||||
|
||||
# chdir to the base directory containing the logs
|
||||
pushd "$LOG_BASE_DIR" >/dev/null 2>&1
|
||||
FULL_PWD="$(realpath "$(pwd)")"
|
||||
|
||||
# cleanup - on exit ensure the temporary directory is removed
|
||||
function cleanup {
|
||||
popd >/dev/null 2>&1
|
||||
if ! rm -rf "$WORKDIR"; then
|
||||
echo "Failed to remove temporary directory '$WORKDIR'" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [ -d "$WORKDIR" ]; then
|
||||
# ensure that if we "grabbed a lock", we release it (works for clean exit, SIGTERM, and SIGINT/Ctrl-C)
|
||||
trap "cleanup" EXIT
|
||||
|
||||
# year month day type hour.0 min.0 sec.0 hour.1 min.1 sec.1
|
||||
PATTERN='(\./)?([0-9]+)-([0-9]+)-([0-9]+)/(.+)\.([0-9]+):([0-9]+):([0-9]+)-([0-9]+):([0-9]+):([0-9]+)\.log\.gz$'
|
||||
|
||||
# find and unzip the compressed zeek logs below this directory into temporary subdirectories that make sense
|
||||
for GZ_LOG_FILE in $(find . -type f -name "*.log.gz"); do
|
||||
GZ_LOG_FILE_SUBDIR="$(dirname "$GZ_LOG_FILE")"
|
||||
GZ_LOG_FILE_DESTDIR="$WORKDIR"/"$GZ_LOG_FILE_SUBDIR"
|
||||
mkdir -p "$GZ_LOG_FILE_DESTDIR"
|
||||
if [[ $GZ_LOG_FILE =~ $PATTERN ]]; then
|
||||
LOG_TYPE=${BASH_REMATCH[5]}
|
||||
DIR_DATE=${BASH_REMATCH[2]}_${BASH_REMATCH[3]}_${BASH_REMATCH[4]}_${BASH_REMATCH[6]}
|
||||
LOG_BASENAME="$(echo "$LOG_TYPE" | awk '{print tolower($0)}')".log
|
||||
DIR_COUNT=0
|
||||
while [[ true ]]; do
|
||||
DEST_DIR="$WORKDIR"/$DIR_DATE.$(printf %02d $DIR_COUNT)
|
||||
DEST_FILE="$DEST_DIR"/"$LOG_BASENAME"
|
||||
if [[ -e "$DEST_FILE" ]]; then
|
||||
DIR_COUNT=$((DIR_COUNT+1))
|
||||
else
|
||||
break
|
||||
fi
|
||||
done
|
||||
mkdir -p "$DEST_DIR"/
|
||||
gunzip --to-stdout "$GZ_LOG_FILE" > "$DEST_FILE"
|
||||
if [[ -n $VERBOSE_FLAG ]]; then
|
||||
FILE_TYPE="$(file -b "$DEST_FILE")"
|
||||
FILE_SIZE="$(fsize $(stat --printf="%s" "$DEST_FILE"))"
|
||||
echo "$DEST_FILE: $FILE_TYPE ($FILE_SIZE)"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# package up all of the log files in their respective directories under our temporary one
|
||||
REPACKAGED_LOGS_TARBALL="$FULL_PWD"/zeek-logs-compressed-$(date +'%Y%m%d_%H%M%S').tar.gz
|
||||
tar -c -z $VERBOSE_FLAG -C "$WORKDIR" -f $REPACKAGED_LOGS_TARBALL .
|
||||
if [[ -n $VERBOSE_FLAG ]]; then
|
||||
FILE_TYPE="$(file -b "$REPACKAGED_LOGS_TARBALL")"
|
||||
FILE_SIZE="$(fsize $(stat --printf="%s" "$REPACKAGED_LOGS_TARBALL"))"
|
||||
echo "$REPACKAGED_LOGS_TARBALL: $FILE_TYPE ($FILE_SIZE)"
|
||||
else
|
||||
echo "$REPACKAGED_LOGS_TARBALL"
|
||||
fi
|
||||
|
||||
fi
|
||||
1
Vagrant/resources/malcolm/scripts/restart
Symbolic link
1
Vagrant/resources/malcolm/scripts/restart
Symbolic link
@@ -0,0 +1 @@
|
||||
./control.py
|
||||
1
Vagrant/resources/malcolm/scripts/start
Symbolic link
1
Vagrant/resources/malcolm/scripts/start
Symbolic link
@@ -0,0 +1 @@
|
||||
./control.py
|
||||
1
Vagrant/resources/malcolm/scripts/status
Symbolic link
1
Vagrant/resources/malcolm/scripts/status
Symbolic link
@@ -0,0 +1 @@
|
||||
control.py
|
||||
1
Vagrant/resources/malcolm/scripts/stop
Symbolic link
1
Vagrant/resources/malcolm/scripts/stop
Symbolic link
@@ -0,0 +1 @@
|
||||
./control.py
|
||||
1
Vagrant/resources/malcolm/scripts/wipe
Symbolic link
1
Vagrant/resources/malcolm/scripts/wipe
Symbolic link
@@ -0,0 +1 @@
|
||||
./control.py
|
||||
Reference in New Issue
Block a user