100 lines
4.1 KiB
Markdown
100 lines
4.1 KiB
Markdown
# Log Collection Lab
|
|
|
|
## Overview
|
|
This repository creates a two node opensearch cluster in an fully automated way.
|
|
Certificates created (no demo certs) and default passwords changed.
|
|
It includes demo configurations for logstash and filebeats.
|
|
|
|
Full feature list:
|
|
- vagrant to download and create an Ubuntu VM and installs docker
|
|
- setup container to create a CA and certificates for the services
|
|
- two node opensearch cluster
|
|
- uses your own certificates
|
|
- default passwords changed
|
|
- opensearch-dashboard
|
|
- creates index pattern automatically in global tenant
|
|
- Grafana
|
|
- creates datasource automatically
|
|
- traefik reverse proxy for opensearch-dashboard
|
|
- mdns responder to allow connection by hostname instead of IP
|
|
- beats logstash container to other computers in your lab can send information (winlogbeat, auditbeat, or packetbeat)
|
|
- log receiver for Cisco ASA, Cisco IOS, Checkpoint, Snort, CEF, Syslog, and Netflow
|
|
- cron container to download json information periodically from API including filebeat/logstash pipeline
|
|
|
|
The repository was created to give you a starting point for your own opensearch installation.
|
|
It shows you how to change common settings and replace default passwords and certificates.
|
|
The demo configuration give you a start into file injections, syslog, grok, beats, logstash configs, certificate creation, docker, vagrant, and more.
|
|
|
|
## Requirements
|
|
Install virtualbox and vagrant on your computer.
|
|
|
|
## Start
|
|
Run
|
|
```
|
|
vagrant up
|
|
````
|
|
in the directory.
|
|
|
|
## Opensearch Dashboard login
|
|
URL: https://opensearch.local (or http://192.168.57.2:5601)
|
|
Username: admin
|
|
Password: vagrant
|
|
|
|
## Grafana Dashboard login
|
|
URL: https://grafana.local
|
|
Username: admin
|
|
Password: vagrant
|
|
|
|
# Network
|
|
The logger virtual machine has three network interfaces.
|
|
1. NAT
|
|
2. private network with static IP 192.168.57.2 (only reachable from your host)
|
|
3. bridged network with dhcp
|
|
|
|
you can send beats from other hosts to the bridged IP address.
|
|
|
|
## Beats
|
|
- install a beats collector (https://www.elastic.co/beats/) on your computer
|
|
- winlogbeat, auditbeat, or packetbeat are easy to get started
|
|
- change output settings in the configuration file to:
|
|
```
|
|
output.logstash:
|
|
hosts: ["192.168.57.2:5044"]
|
|
```
|
|
You will need to remove the output.elasticsearch section.
|
|
- information will be stored in opensearch-logstash-beats-$date
|
|
- you just need to create the index pattern
|
|
|
|
## Password changes
|
|
All passwords are set to "vagrant" in this repository.
|
|
The password hashes are stored in internal_users.yml and the logstash clear text password is in the .env file (used by logstash containers)
|
|
If you want to change the password your need to replace the hashes and tell opensearch to read the configuration.
|
|
The securityadmin command must be executed in both opensearch nodes
|
|
```
|
|
docker exec -it opensearch-node1 /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh --clustername opensearch-cluster --configdir /usr/share/opensearch/config/opensearch-security -cacert /usr/share/opensearch/config/certs/opensearch-ca.pem -key /usr/share/opensearch/config/certs/opensearch-admin.key -cert /usr/share/opensearch/config/certs/opensearch-admin.pem -h `cat /etc/hostname`
|
|
|
|
docker exec -it opensearch-node2 /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh --clustername opensearch-cluster --configdir /usr/share/opensearch/config/opensearch-security -cacert /usr/share/opensearch/config/certs/opensearch-ca.pem -key /usr/share/opensearch/config/certs/opensearch-admin.key -cert /usr/share/opensearch/config/certs/opensearch-admin.pem -h `cat /etc/hostname`
|
|
```
|
|
|
|
# Troubleshooting
|
|
## Docker
|
|
```
|
|
vagrant ssh
|
|
```
|
|
-> logs you into the VM
|
|
|
|
```
|
|
sudo -s && cd /vagrant && docker-compose logs -f
|
|
```
|
|
-> all files are mapped inth this folder of the VM. you can use all docker and docker-compose commands as usual (ps, exec, ...)
|
|
|
|
## Elasticsearch
|
|
GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state
|
|
|
|
GET _cluster/allocation/explain
|
|
|
|
GET _cluster/settings?flat_settings=true&include_defaults=true
|
|
|
|
PUT _cluster/settings
|
|
{ "persistent" : { "cluster.routing.allocation.enable" : "all" } }
|