Merge pull request #262 from clong/caldera2

Remove Caldera entirely, reduce verbosity

README.md

@@ -28,7 +28,6 @@ NOTE: This lab has not been hardened in any way and runs with default vagrant cr
 * Powershell transcript logging is enabled. All logs are saved to `\\wef\pslogs`
 * osquery comes installed on each host and is pre-configured to connect to a [Fleet](https://kolide.co/fleet) server via TLS. Fleet is preconfigured with the configuration from [Palantir's osquery Configuration](https://github.com/palantir/osquery-configuration)
 * Sysmon is installed and configured using SwiftOnSecurity’s open-sourced configuration
-* Mitre's [Caldera](https://github.com/mitre/caldera) server is built on the logger host and the Caldera agent gets pre-installed on all Windows hosts
 * All autostart items are logged to Windows Event Logs via [AutorunsToWinEventLog](https://github.com/palantir/windows-event-forwarding/tree/master/AutorunsToWinEventLog)
 * SMBv1 Auditing is enabled
 
@@ -93,7 +92,7 @@ $ packer build --only=[vmware|virtualbox]-iso windows_2016.json
 4. cd into the Vagrant directory: `cd ../Vagrant`
 5. Install the Vagrant-Reload plugin: `vagrant plugin install vagrant-reload`
 
-6. Ensure you are in the Vagrant folrder and run `vagrant up`. This command will do the following:
+6. Ensure you are in the Vagrant folder and run `vagrant up`. This command will do the following:
   * Provision the logger host. This host will run the [Fleet](https://kolide.co/fleet) osquery manager and a fully featured pre-configured Splunk instance.
   * Provision the DC host and configure it as a Domain Controller
   * Provision the WEF host and configure it as a Windows Event Collector in the Servers OU
@@ -101,7 +100,6 @@ $ packer build --only=[vmware|virtualbox]-iso windows_2016.json
 
 7. Navigate to https://192.168.38.105:8000 in a browser to access the Splunk instance on logger. Default credentials are admin:changeme (you will have the option to change them on the next screen)
 8. Navigate to https://192.168.38.105:8412 in a browser to access the Fleet server on logger. Default credentials are admin:admin123#. Query packs are pre-configured with queries from [palantir/osquery-configuration](https://github.com/palantir/osquery-configuration).
-9. Navigate to https://192.168.38.105:8888 in a browser to access the Caldera server on logger. Default credentials are admin:caldera.
 
 ## Basic Vagrant Usage
 Vagrant commands must be run from the "Vagrant" folder.
@@ -122,9 +120,8 @@ Vagrant commands must be run from the "Vagrant" folder.
 ## Lab Information
 * Domain Name: windomain.local
 * Admininstrator login: vagrant:vagrant
-* Fleet login: https://192.168.38.105:8412 - admin@detectionlab.network:admin123#
+* Fleet login: https://192.168.38.105:8412 - admin:admin123#
 * Splunk login: https://192.168.38.105:8000 - admin:changeme
-* Caldera login: https://192.168.38.105:8888 - admin:caldera
 * MS ATA login: https://192.168.38.103 - wef\vagrant:vagrant
 
 ## Lab Hosts
@@ -155,7 +152,6 @@ Vagrant commands must be run from the "Vagrant" folder.
 * Logger - Ubuntu 16.04
   * Splunk Enterprise
   * Fleet osquery Manager
-  * Mitre's Caldera Server
   * Bro
   * Suricata
 
@@ -174,7 +170,6 @@ suricata | Suricata IDS logs
   * Sysmon
   * osquery
   * AutorunsToWinEventLog
-  * Caldera Agent
   * Process Monitor
   * Process Explorer
   * PsExec
@@ -184,6 +179,7 @@ suricata | Suricata IDS logs
   * WinRar
   * Mimikatz
   * Wireshark
+  * Powersploit
 
 ## Applied GPOs
 * [Custom Event Channel Permissions](https://rawgit.com/clong/DetectionLab/master/Vagrant/resources/GPO/reports/Custom%20Event%20Channel%20Permissions.htm)
@@ -255,7 +251,6 @@ A sizable percentage of this code was borrowed and adapted from [Stefan Scherer]
 * [Splunk](https://www.splunk.com)
 * [osquery](https://osquery.io)
 * [Fleet](https://kolide.co/fleet)
-* [Caldera](https://github.com/mitre/caldera)
 * [Windows Event Forwarding for Network Defense](https://medium.com/@palantir/windows-event-forwarding-for-network-defense-cb208d5ff86f)
 * [palantir/windows-event-forwarding](http://github.com/palantir/windows-event-forwarding)
 * [osquery Across the Enterprise](https://medium.com/@palantir/osquery-across-the-enterprise-3c3c9d13ec55)

Terraform/main.tf

@@ -71,14 +71,6 @@ resource "aws_security_group" "logger" {
     cidr_blocks = "${var.ip_whitelist}"
   }
 
-  # Caldera access
-  ingress {
-    from_port   = 8888
-    to_port     = 8888
-    protocol    = "tcp"
-    cidr_blocks = "${var.ip_whitelist}"
-  }
-
   # Allow all traffic from the private subnet
   ingress {
     from_port   = 0
@@ -166,16 +158,13 @@ resource "aws_instance" "logger" {
       "sudo mkdir /home/vagrant/.ssh && sudo cp /home/ubuntu/.ssh/authorized_keys /home/vagrant/.ssh/authorized_keys && sudo chown -R vagrant:vagrant /home/vagrant/.ssh",
       "echo 'vagrant    ALL=(ALL:ALL) NOPASSWD:ALL' | sudo tee -a /etc/sudoers",
       "sudo git clone https://github.com/clong/DetectionLab.git /opt/DetectionLab",
-      "sudo sed -i \"s#sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/g' /etc/apt/sources.list##g\" /opt/DetectionLab/Vagrant/bootstrap.sh",
       "sudo sed -i 's/eth1/eth0/g' /opt/DetectionLab/Vagrant/bootstrap.sh",
       "sudo sed -i 's/ETH1/ETH0/g' /opt/DetectionLab/Vagrant/bootstrap.sh",
       "sudo sed -i 's#/usr/local/go/bin/go get -u#GOPATH=/root/go /usr/local/go/bin/go get -u#g' /opt/DetectionLab/Vagrant/bootstrap.sh",
       "sudo sed -i 's#/vagrant/resources#/opt/DetectionLab/Vagrant/resources#g' /opt/DetectionLab/Vagrant/bootstrap.sh",
       "sudo chmod +x /opt/DetectionLab/Vagrant/bootstrap.sh",
-      "sudo apt-get update",
+      "sudo apt-get -qq update",
       "sudo /opt/DetectionLab/Vagrant/bootstrap.sh",
-      "sudo pip3.6 install --upgrade --force-reinstall pip==9.0.3 && sudo pip3.6 install -r /home/vagrant/caldera/caldera/requirements.txt && sudo pip3.6 install --upgrade pip",
-      "sudo service caldera stop && sudo service caldera start",
     ]
     connection {
       type = "ssh"

Vagrant/Vagrantfile

@@ -48,7 +48,6 @@ Vagrant.configure("2") do |config|
     cfg.vm.provision "shell", path: "scripts/install-redteam.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-choco-extras.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/install-caldera-agent.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-sysinternals.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-ou.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-wef-gpo.ps1", privileged: false
@@ -104,7 +103,6 @@ Vagrant.configure("2") do |config|
     cfg.vm.provision "shell", path: "scripts/install-redteam.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-choco-extras.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/install-caldera-agent.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-sysinternals.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-pslogstranscriptsshare.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-autorunstowineventlog.ps1", privileged: false
@@ -154,7 +152,6 @@ Vagrant.configure("2") do |config|
     cfg.vm.provision "shell", path: "scripts/install-redteam.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-choco-extras.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/install-caldera-agent.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-inputsconf.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-sysinternals.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-autorunstowineventlog.ps1", privileged: false

Vagrant/Vagrantfile_Minimum

@@ -6,13 +6,6 @@ Vagrant.configure("2") do |config|
     config.vm.provision :shell, path: "bootstrap.sh"
     cfg.vm.network :private_network, ip: "192.168.38.105", gateway: "192.168.38.1", dns: "8.8.8.8"
 
-    cfg.vm.provider "vmware_fusion" do |v, override|
-      v.vmx["displayname"] = "logger"
-      v.memory = 2048
-      v.cpus = 1
-      v.gui = true
-    end
-
     cfg.vm.provider "vmware_desktop" do |v, override|
       v.vmx["displayname"] = "logger"
       v.memory = 4096
@@ -33,13 +26,9 @@ Vagrant.configure("2") do |config|
   end
 
   config.vm.define "dc" do |cfg|
-    cfg.vm.box = "../Boxes/windows_2016_virtualbox.box"
+    cfg.vm.box = "detectionlab/win2016"
     cfg.vm.hostname = "dc"
     cfg.vm.boot_timeout = 600
-    # use the plaintext WinRM transport and force it to use basic authentication.
-    # NB this is needed because the default negotiate transport stops working
-    #    after the domain controller is installed.
-    #    see https://groups.google.com/forum/#!topic/vagrant-up/sZantuCM0q4
     cfg.winrm.transport = :plaintext
     cfg.vm.communicator = "winrm"
     cfg.winrm.basic_auth_only = true
@@ -57,28 +46,19 @@ Vagrant.configure("2") do |config|
     cfg.vm.provision "shell", path: "scripts/install-inputsconf.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-utilities.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/install-caldera-agent.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-sysinternals.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-ou.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-wef-gpo.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-powershelllogging.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-AuditingPolicyGPOs.ps1", privileged: false
+    cfg.vm.provision "shell", path: "scripts/configure-rdp-user-gpo.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-autorunstowineventlog.ps1", privileged: false
     cfg.vm.provision "shell", inline: 'wevtutil el | Select-String -notmatch "Microsoft-Windows-LiveId" | Foreach-Object {wevtutil cl "$_"}', privileged: false
     cfg.vm.provision "shell", inline: "Set-SmbServerConfiguration -AuditSmb1Access $true -Force", privileged: false
 
-    cfg.vm.provider "vmware_fusion" do |v, override|
-      override.vm.box = "../Boxes/windows_2016_vmware.box"
-      v.vmx["displayname"] = "dc.windomain.local"
-      v.memory = 2560
-      v.cpus = 2
-      v.gui = true
-    end
-
     cfg.vm.provider "vmware_desktop" do |v, override|
-      override.vm.box = "../Boxes/windows_2016_vmware.box"
       v.vmx["displayname"] = "dc.windomain.local"
-      v.memory = 2560
+      v.memory = 3072
       v.cpus = 2
       v.gui = true
       v.enable_vmrun_ip_lookup = false
@@ -88,7 +68,7 @@ Vagrant.configure("2") do |config|
       vb.gui = true
       vb.name = "dc.windomain.local"
       vb.default_nic_type = "82545EM"
-      vb.customize ["modifyvm", :id, "--memory", 2560]
+      vb.customize ["modifyvm", :id, "--memory", 3072]
       vb.customize ["modifyvm", :id, "--cpus", 2]
       vb.customize ["modifyvm", :id, "--vram", "32"]
       vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]
@@ -97,7 +77,7 @@ Vagrant.configure("2") do |config|
   end
 
   config.vm.define "wef" do |cfg|
-    cfg.vm.box = "../Boxes/windows_2016_virtualbox.box"
+    cfg.vm.box = "detectionlab/win2016"
     cfg.vm.hostname = "wef"
     cfg.vm.boot_timeout = 600
     cfg.vm.communicator = "winrm"
@@ -119,22 +99,12 @@ Vagrant.configure("2") do |config|
     cfg.vm.provision "shell", path: "scripts/install-inputsconf.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-utilities.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/install-caldera-agent.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-sysinternals.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/configure-pslogstranscriptsshare.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-autorunstowineventlog.ps1", privileged: false
     cfg.vm.provision "shell", inline: "Set-SmbServerConfiguration -AuditSmb1Access $true -Force", privileged: false
 
-    cfg.vm.provider "vmware_fusion" do |v, override|
-      override.vm.box = "../Boxes/windows_2016_vmware.box"
-      v.vmx["displayname"] = "wef.windomain.local"
-      v.memory = 2048
-      v.cpus = 2
-      v.gui = true
-    end
-
     cfg.vm.provider "vmware_desktop" do |v, override|
-      override.vm.box = "../Boxes/windows_2016_vmware.box"
       v.vmx["displayname"] = "wef.windomain.local"
       v.memory = 2048
       v.cpus = 2
@@ -155,7 +125,7 @@ Vagrant.configure("2") do |config|
   end
 
   config.vm.define "win10" do |cfg|
-    cfg.vm.box = "../Boxes/windows_10_virtualbox.box"
+    cfg.vm.box = "detectionlab/win10"
     cfg.vm.hostname = "win10"
     cfg.vm.boot_timeout = 600
     cfg.vm.communicator = "winrm"
@@ -165,34 +135,21 @@ Vagrant.configure("2") do |config|
     cfg.vm.network :private_network, ip: "192.168.38.104", gateway: "192.168.38.1", dns: "192.168.38.102"
 
     cfg.vm.provision "shell", path: "scripts/fix-second-network.ps1", privileged: false, args: "-ip 192.168.38.104 -dns 192.168.38.102"
+    cfg.vm.provision "shell", path: "scripts/MakeWindows10GreatAgain.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/provision.ps1", privileged: false
     cfg.vm.provision "reload"
     cfg.vm.provision "shell", path: "scripts/provision.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/download_palantir_wef.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/download_palantir_osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/MakeWindows10GreatAgain.ps1", privileged: false
     cfg.vm.provision "shell", inline: 'wevtutil el | Select-String -notmatch "Microsoft-Windows-LiveId" | Foreach-Object {wevtutil cl "$_"}', privileged: false
     cfg.vm.provision "shell", path: "scripts/install-splunkuf.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-utilities.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-osquery.ps1", privileged: false
-    cfg.vm.provision "shell", path: "scripts/install-caldera-agent.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-inputsconf.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-sysinternals.ps1", privileged: false
     cfg.vm.provision "shell", path: "scripts/install-autorunstowineventlog.ps1", privileged: false
 
-    cfg.vm.provider "vmware_fusion" do |v, override|
-      override.vm.box = "../Boxes/windows_10_vmware.box"
-      v.vmx["displayname"] = "win10.windomain.local"
-      v.vmx["gui.fullscreenatpoweron"] = "FALSE"
-      v.vmx["gui.viewModeAtPowerOn"] = "windowed"
-      v.vmx["gui.fitguestusingnativedisplayresolution"] = "FALSE"
-      v.memory = 2048
-      v.cpus = 1
-      v.gui = true
-    end
-
     cfg.vm.provider "vmware_desktop" do |v, override|
-      override.vm.box = "../Boxes/windows_10_vmware.box"
       v.vmx["displayname"] = "win10.windomain.local"
       v.vmx["gui.fullscreenatpoweron"] = "FALSE"
       v.vmx["gui.viewModeAtPowerOn"] = "windowed"

Vagrant/bootstrap.sh

@@ -5,32 +5,19 @@ echo "apt-fast apt-fast/maxdownloads string 10" | debconf-set-selections;
 echo "apt-fast apt-fast/dlflag boolean true" | debconf-set-selections;
 sed -i "2ideb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt xenial-backports main restricted universe multiverse\ndeb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse" /etc/apt/sources.list
 
-install_mongo_db_apt_key() {
-  # Install key and apt source for MongoDB
-  apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
-  echo "deb http://repo.mongodb.org/apt/ubuntu $(lsb_release -sc)/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
-}
-
-install_python_apt_source() {
-  # Install apt source for Python3.6
-  add-apt-repository -y ppa:jonathonf/python-3.6
-  add-apt-repository -y ppa:apt-fast/stable
-}
-
 apt_install_prerequisites() {
+  # Add repository for apt-fast
+  add-apt-repository -y ppa:apt-fast/stable
   # Install prerequisites and useful tools
   echo "Running apt-get update..."
   apt-get -qq update
   apt-get -qq install -y apt-fast
   echo "Running apt-fast install..."
-  apt-fast -qq install -y jq whois build-essential git docker docker-compose unzip mongodb-org python3.6 python3.6-dev
+  apt-fast -qq install -y jq whois build-essential git docker docker-compose unzip
-  # Install pip for Python 3.6
-  echo "Installing Pip3.6..."
-  curl https://bootstrap.pypa.io/get-pip.py | sudo -H python3.6
 }
 
 test_prerequisites() {
-  for package in jq whois build-essential git docker docker-compose unzip mongodb-org python3.6 python3.6-dev
+  for package in jq whois build-essential git docker docker-compose unzip
   do
     echo "[TEST] Validating that $package is correctly installed..."
     # Loop through each package using dpkg
@@ -47,25 +34,6 @@ test_prerequisites() {
       echo "[+] $package was successfully installed!"
     fi
   done
-
-  # One-off support for packages which aren't installed via dpkg
-  for package in "pip3.6"
-  do
-    echo "[TEST] Validating that $package is correctly installed..."
-    # Loop through each package using which
-    if ! which $package > /dev/null; then
-      # If which returns a non-zero return code, try to re-install the package
-      echo "[-] $package was not found. Attempting to reinstall."
-      curl https://bootstrap.pypa.io/get-pip.py | sudo -H python3.6
-      if ! which $package > /dev/null; then
-        # If the reinstall fails, give up
-        echo "[X] Unable to install $package even after a retry. Exiting."
-        exit 1
-      fi
-    else
-      echo "[+] $package was successfully installed!"
-    fi
-  done
 }
 
 fix_eth1_static_ip() {
@@ -113,9 +81,22 @@ install_splunk() {
     echo "Installing Splunk..."
     # Get Splunk.com into the DNS cache. Sometimes resolution randomly fails during wget below
     dig @8.8.8.8 splunk.com
-    # Download Splunk
+    mkdir splunk
-    wget --progress=bar:force -O splunk-7.2.5.1-962d9a8e1586-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.5.1&product=splunk&filename=splunk-7.2.5.1-962d9a8e1586-linux-2.6-amd64.deb&wget=true'
+
-    dpkg -i splunk-7.2.5.1-962d9a8e1586-linux-2.6-amd64.deb
+    # Try to resolve the latest version of Splunk by parsing the HTML on the downloads page
+    LATEST_SPLUNK=$(curl https://www.splunk.com/en_us/download/splunk-enterprise.html | grep -i deb | grep -Eo "data-link=\"................................................................................................................................" | cut -d '"' -f 2)
+    # Sanity check what was returned from the auto-parse attempt
+    echo "Attempting to autoresolve the latest version of Splunk..."
+    if [[ "$(echo $LATEST_SPLUNK | grep -c "^https:")" -eq 1 ]] && [[ "$(echo $LATEST_SPLUNK | grep -c "\.deb$")" -eq 1 ]]; then
+      echo "The URL to the latest Splunk version was automatically resolved as: $LATEST_SPLUNK"
+      echo "Attempting to download..."
+      wget --progress=bar:force -P splunk/ "$LATEST_SPLUNK"
+    else
+      echo "Unable to auto-resolve the latest Splunk version. Falling back to hardcoded URL..."
+      # Download Hardcoded Splunk
+      wget --progress=bar:force -O splunk/splunk-7.2.6-c0bf0f679ce9-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.6&product=splunk&filename=splunk-7.2.6-c0bf0f679ce9-linux-2.6-amd64.deb&wget=true'
+    fi
+    dpkg -i splunk/*.deb
     /opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd changeme
     /opt/splunk/bin/splunk add index wineventlog -auth 'admin:changeme'
     /opt/splunk/bin/splunk add index osquery -auth 'admin:changeme'
@@ -192,12 +173,12 @@ download_palantir_osquery_config() {
 }
 
 import_osquery_config_into_fleet() {
-  wget --progress=bar:force https://github.com/kolide/fleet/releases/download/2.0.1/fleet_2.0.1.zip
+  wget --progress=bar:force https://github.com/kolide/fleet/releases/download/2.1.1/fleet_2.1.1.zip
-  unzip fleet_2.0.1.zip -d fleet_2.0.1
+  unzip fleet_2.1.1.zip -d fleet_2.1.1
-  cp fleet_2.0.1/linux/fleetctl /usr/local/bin/fleetctl && chmod +x /usr/local/bin/fleetctl
+  cp fleet_2.1.1/linux/fleetctl /usr/local/bin/fleetctl && chmod +x /usr/local/bin/fleetctl
   fleetctl config set --address https://192.168.38.105:8412
   fleetctl config set --tls-skip-verify true
-  fleetctl setup --email admin@detectionlab.network --password 'admin123#' --org-name DetectionLab
+  fleetctl setup --email admin@detectionlab.network --username admin --password 'admin123#' --org-name DetectionLab
   fleetctl login --email admin@detectionlab.network --password 'admin123#'
 
   # Use fleetctl to import YAML files
@@ -212,36 +193,6 @@ import_osquery_config_into_fleet() {
   /opt/splunk/bin/splunk add monitor "/home/vagrant/kolide-quickstart/osquery_status" -index osquery-status -sourcetype 'osquery:status' -auth 'admin:changeme'
 }
 
-install_caldera() {
-  if [ -f "/lib/systemd/system/caldera.service" ]; then
-    echo "Caldera is already installed... Skipping"
-  else
-    # Install Mitre's Caldera
-    echo "Installing Caldera..."
-    cd /home/vagrant || exit
-    git clone https://github.com/mitre/caldera.git
-    cd /home/vagrant/caldera/caldera || exit
-    pip3.6 install -r requirements.txt
-
-    # Add a Systemd service for MongoDB
-    # https://www.howtoforge.com/tutorial/install-mongodb-on-ubuntu-16.04/
-    cp /vagrant/resources/caldera/mongod.service /lib/systemd/system/mongod.service
-    # Create Systemd service for Caldera
-    cp /vagrant/resources/caldera/caldera.service /lib/systemd/system/caldera.service
-    # Enable replication
-    echo 'replication:
-    replSetName: caldera' >> /etc/mongod.conf
-    service mongod start
-    systemctl enable mongod.service
-    cd /home/vagrant/caldera || exit
-    mkdir -p dep/crater/crater
-    wget --progress=bar:force https://github.com/mitre/caldera-crater/releases/download/v0.1.0/CraterMainWin8up.exe -O /home/vagrant/caldera/dep/crater/crater/CraterMain.exe
-    cp /vagrant/resources/caldera/cert.pem /vagrant/resources/caldera/key.pem /vagrant/resources/caldera/settings.yml /home/vagrant/caldera/caldera/conf
-    service caldera start
-    systemctl enable caldera.service
-  fi
-}
-
 install_bro() {
   # Environment variables
   NODECFG=/opt/bro/etc/node.cfg
@@ -423,8 +374,6 @@ test_suricata_prerequisites() {
 }
 
 main() {
-  install_mongo_db_apt_key
-  install_python_apt_source
   apt_install_prerequisites
   test_prerequisites
   fix_eth1_static_ip
@@ -433,7 +382,6 @@ main() {
   install_fleet
   download_palantir_osquery_config
   import_osquery_config_into_fleet
-  install_caldera
   install_suricata
   install_bro
 }

Vagrant/resources/caldera/caldera.service

@@ -1,11 +0,0 @@
-[Unit]
-Description=My Script Service
-After=multi-user.target
-
-[Service]
-Type=idle
-WorkingDirectory=/home/vagrant/caldera/caldera
-ExecStart=/usr/bin/python3.6 caldera.py
-
-[Install]
-WantedBy=multi-user.target

Vagrant/resources/caldera/cert.pem

@@ -1,20 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDVTCCAj2gAwIBAgIJAKUxnMnJz4CPMA0GCSqGSIb3DQEBCwUAMEExDzANBgNV
-BAMMBmxvZ2dlcjEQMA4GA1UECwwHU2VydmVyczEPMA0GA1UECgwGbG9nZ2VyMQsw
-CQYDVQQGEwJVUzAeFw0xOTAzMjcwMzU3NThaFw0yOTAzMjYwMzU3NThaMEExDzAN
-BgNVBAMMBmxvZ2dlcjEQMA4GA1UECwwHU2VydmVyczEPMA0GA1UECgwGbG9nZ2Vy
-MQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKL/
-7cDUXrSac8oILHLN3iKtuhoLttPrVi+GuYsdfKer7D9M9DrkGAvQ+qHLpHgg+omY
-ZIK4kDjS26eDKP2SGTHjeYwlmcJPnII6YU8vYdzyvi4eLJjI+lMO8BYd2yL2PVJX
-W8ScY/2MoC8LX31A8maGDW+4E6bu3ItAEchh4EWRj8V4PqCDJH1XnY18A0N0QEDV
-Osleb2dFWvXtyFmSu0OsqiMxBzlPbf/yl71Q1VjrRUT+alDJNfF3ewctshvx4hV8
-qDwpATR9GPfPKV6w1lj0a+81Qpwy6vOLve3yCV02lWe1TaCjL2otGbTspujbulkR
-DGy6MC9/9i1CRkEIBj0CAwEAAaNQME4wHQYDVR0OBBYEFKGWfBB6O2ktq/a4frXy
-LKxZNjngMB8GA1UdIwQYMBaAFKGWfBB6O2ktq/a4frXyLKxZNjngMAwGA1UdEwQF
-MAMBAf8wDQYJKoZIhvcNAQELBQADggEBAIgLaPeFJS/bHzITf62RgDbBEU3zaeOg
-72xiFHWd6uIA9SCHGHuKhbpzLOqBtoSGZlejI75/QDwAiwpu/cd9BWx9h9S/f0Zf
-fZnA/ngPojhnaHYFIn8165z/XSJwNvDhVh1oYqMcfTPgeSlbRdPTwAjxdYsJ4nik
-4xriKHV6WQ9ixFwOk9ROYg1M/OJ2lmOi23NFODKVP0LFPAqZEX7evKJatWQiJ2tF
-iUAm+4xvkGVJfPd6hShYEsXDjTJC+fVFgsPTWhaHGnLjiLORJLgvd/E6wIGGzvx7
-fIw1x7LxMmhlWa74yyGZKAMUOlMh4uBAYR3JhsVYl2LNSK9HwZYnfXc=
------END CERTIFICATE-----

Vagrant/resources/caldera/conf.yml

@@ -1,24 +0,0 @@
-url_root: https://logger:8888
-verify_hostname: false
-cert: |
-  -----BEGIN CERTIFICATE-----
-  MIIDVTCCAj2gAwIBAgIJAKUxnMnJz4CPMA0GCSqGSIb3DQEBCwUAMEExDzANBgNV
-  BAMMBmxvZ2dlcjEQMA4GA1UECwwHU2VydmVyczEPMA0GA1UECgwGbG9nZ2VyMQsw
-  CQYDVQQGEwJVUzAeFw0xOTAzMjcwMzU3NThaFw0yOTAzMjYwMzU3NThaMEExDzAN
-  BgNVBAMMBmxvZ2dlcjEQMA4GA1UECwwHU2VydmVyczEPMA0GA1UECgwGbG9nZ2Vy
-  MQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKL/
-  7cDUXrSac8oILHLN3iKtuhoLttPrVi+GuYsdfKer7D9M9DrkGAvQ+qHLpHgg+omY
-  ZIK4kDjS26eDKP2SGTHjeYwlmcJPnII6YU8vYdzyvi4eLJjI+lMO8BYd2yL2PVJX
-  W8ScY/2MoC8LX31A8maGDW+4E6bu3ItAEchh4EWRj8V4PqCDJH1XnY18A0N0QEDV
-  Osleb2dFWvXtyFmSu0OsqiMxBzlPbf/yl71Q1VjrRUT+alDJNfF3ewctshvx4hV8
-  qDwpATR9GPfPKV6w1lj0a+81Qpwy6vOLve3yCV02lWe1TaCjL2otGbTspujbulkR
-  DGy6MC9/9i1CRkEIBj0CAwEAAaNQME4wHQYDVR0OBBYEFKGWfBB6O2ktq/a4frXy
-  LKxZNjngMB8GA1UdIwQYMBaAFKGWfBB6O2ktq/a4frXyLKxZNjngMAwGA1UdEwQF
-  MAMBAf8wDQYJKoZIhvcNAQELBQADggEBAIgLaPeFJS/bHzITf62RgDbBEU3zaeOg
-  72xiFHWd6uIA9SCHGHuKhbpzLOqBtoSGZlejI75/QDwAiwpu/cd9BWx9h9S/f0Zf
-  fZnA/ngPojhnaHYFIn8165z/XSJwNvDhVh1oYqMcfTPgeSlbRdPTwAjxdYsJ4nik
-  4xriKHV6WQ9ixFwOk9ROYg1M/OJ2lmOi23NFODKVP0LFPAqZEX7evKJatWQiJ2tF
-  iUAm+4xvkGVJfPd6hShYEsXDjTJC+fVFgsPTWhaHGnLjiLORJLgvd/E6wIGGzvx7
-  fIw1x7LxMmhlWa74yyGZKAMUOlMh4uBAYR3JhsVYl2LNSK9HwZYnfXc=
-  -----END CERTIFICATE-----
-logging_level: debug

Vagrant/resources/caldera/key.pem

@@ -1,28 +0,0 @@
------BEGIN PRIVATE KEY-----
-MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCi/+3A1F60mnPK
-CCxyzd4irboaC7bT61YvhrmLHXynq+w/TPQ65BgL0Pqhy6R4IPqJmGSCuJA40tun
-gyj9khkx43mMJZnCT5yCOmFPL2Hc8r4uHiyYyPpTDvAWHdsi9j1SV1vEnGP9jKAv
-C199QPJmhg1vuBOm7tyLQBHIYeBFkY/FeD6ggyR9V52NfANDdEBA1TrJXm9nRVr1
-7chZkrtDrKojMQc5T23/8pe9UNVY60VE/mpQyTXxd3sHLbIb8eIVfKg8KQE0fRj3
-zylesNZY9GvvNUKcMurzi73t8gldNpVntU2goy9qLRm07Kbo27pZEQxsujAvf/Yt
-QkZBCAY9AgMBAAECggEAchAQKyExyMZQgq9nk0r2yGOz2GgcH34OyKFJk80lnxvS
-7ZUHT3KwVQx/Zttfk2TYjZLjuwo2el0rsrQ46gDjuqKJOwZQ68XUKtl5oPTyUbBY
-X2NkJ+J0Mc1BOujXoUaZMNbK8/HQkuxmZ6VYxAi/1/i0zFoO/oEmOheymfvL+kG9
-Yi9o7DR8o8yHT2XAJM3sdRLb19/BdquB8vcXYvHY/X2+2uLU4vDycHZ9bFiRWfMK
-bpgzH0XRsfz5/9nV42gUhPZWbKJYZnauYI4k603i5uT165chkkv4HJld5mgUMZel
-U58zFE9LwGOKuQJbYkvKDj4G+vF4i9ONRMg5jNhO4QKBgQDPRALjFSFoTd5wOjef
-Z2n5P307DPyix+GxbE1VRvsKpWNW/DYmVHrLX9cyZPpVPwcAla11gSoSOQ4AW3F4
-8uVrL86FTPI3h4nCZje6NvLlzNvI7zmNgCVHLrtp16sub7qxRTuLJuO8V0Penka8
-dpRI1npaetvwuVjNjw98NdioWQKBgQDJU2hKEbCmX+h9IfuKdOCT5gMYjii1ndC4
-uUwrUf+Fy9Z0aZIIDa/TKjmVnUEd+2C4AGdWIc0p/ApzKId9o9f0RUTFv0x2fKGt
-wQUE4BuvHBTQY++N6lYSEVBHOicoITbKfTQyiHimzgg19JjRaxPEoI7zla7euOtz
-wD8RZisQhQKBgQCoqP14cG95AzSpOpacvnh7/PAZJ15F0EWWLT1Y0aPja+ZEmIQv
-+Lrb2CezuHwE9n7qOOKoJe1oWslL/R1/gJiz+nkh/BV2nnK5oJhkjeZmnaeUASdi
-QA8Vt6+GV5vS89SGAIKNn53eW1y9QSrqncvY4b869ZMYdpsO2wWM2/xsMQKBgQCU
-cgaszrxlAMTJN3xky+cnxmqz+WY0/AeBnUd3jb+OJQ2cypbdj1H8rt7E3a288LSR
-F+MaFFRVx7zgypqU3nGGcGjUMkir+RuxpWgQ/zoKdPc37OgeyRHrwo5h5pQmt87R
-oV4SgKQNwk81kYP4ri5mRkkBfHXzTYB22eLm7mGEjQKBgB+WaIOMpgQwrewRkU17
-S63jjHso3J4S21OHm748vpx3ize4JpnAkRpfpwZ9Rj+TmyOjgitJNWdV77zdiQkN
-5Dv5ZjsIurrnDXMslBww37mnV7LXyDAAsyaKqSGQuikKusz45FSej82KIfr/zGv+
-WbNZ+sK8mdVZoUSam+6BO9JH
------END PRIVATE KEY-----

Vagrant/resources/caldera/mongod.service

@@ -1,12 +0,0 @@
-[Unit]
-Description=High-performance, schema-free document-oriented database
-After=network.target
-Documentation=https://docs.mongodb.org/manual
-
-[Service]
-User=mongodb
-Group=mongodb
-ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf
-
-[Install]
-WantedBy=multi-user.target

Vagrant/resources/caldera/settings.yml

@@ -1,27 +0,0 @@
-auth:
-  key: !!binary |
-    QfVQNdpJt0br69u4yGYgjJYr8mlKRb8nhkzxd+XkuBt7D/JKeCUmSafJuBFY3w7oXatwSqFYKdWn
-    ZhVklJsZAmoPy3BEr1maTwFFfJtfav6e6JUMfPKP2SXHFW0fsyF3mdxpXtMbRrMTwnQ4TrndAtCs
-    E0jWC9qM/ktTga5UweS3YkDlQpWQHktgPPRC8nLk2dtGkWLuam7lfFCW93mxrNyYAAlbNQPq8/ld
-    DvYD+eWB8tviOh5ZcntmST4tCXXZqk6pGByWV8m7EnBRK4ffIoNmgNV5K7UrWBPEI5ZbNbHx9rSl
-    GNqAiA9T020MJ5qyrCl9mgZTnbmKXiUOcgpr6g==
-crypto:
-  cert: conf/cert.pem
-  key: conf/key.pem
-database:
-  host: localhost
-  key: !!binary |
-    UXFoNGMzdU1kYWJzU195ZDEybUItWnBJbEg5UnNqYndmQWJpcC1xNjFNaz0=
-  port: 27017
-proxy:
-  default:
-    cert: ''
-    http: ''
-    https: ''
-server:
-  host: 0.0.0.0
-  https: true
-  port: 8888
-users:
-- name: admin
-  password: caldera

Vagrant/resources/windows/MenuSettings.xml

@@ -0,0 +1,5 @@
+<?xml version="1.0"?>
+<Settings component="StartMenu" version="4.3.1">
+	<MenuStyle value="Win7"/>
+	<SkipMetro value="1"/>
+</Settings>

Vagrant/resources/windows/classic_shell_win7.reg

@@ -1,4 +0,0 @@
-Windows Registry Editor Version 5.00
-
-[HKEY_LOCAL_MACHINE\SOFTWARE\IvoSoft\ClassicStartMenu]
-"MenuStyle_Default"="Win7"

Vagrant/scripts/install-caldera-agent.ps1

@@ -1,34 +0,0 @@
-# Purpose: Installs the Caldera agent on the host
-
-If (-not (Test-Path 'C:\Program Files\cagent\cagent.exe')) {
-  # Add /etc/hosts entry
-  Add-Content "c:\windows\system32\drivers\etc\hosts" "        192.168.38.105    logger"
-
-  # Make the directory
-  New-Item "c:\Program Files\cagent" -type directory
-
-  # Install Visual Studio 2015 C++ Redistributable
-  choco install -y vcredist2015
-
-  # Download cagent and start the service
-  Write-Host "Downloading Caldera Agent (cagent.exe)"
-  $cagentPath = "C:\Program Files\cagent\cagent.exe"
-  $cagentConfPath = "C:\Program Files\cagent\conf.yml"
-  # GitHub requires TLS 1.2 as of 2/1/2018
-  [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
-  (New-Object System.Net.WebClient).DownloadFile('https://github.com/mitre/caldera-agent/releases/download/v0.1.0/cagent.exe', $cagentPath)
-  # Copy hardocded Caldera config from the Vagrant resources folder
-  Copy-Item "C:\vagrant\resources\caldera\conf.yml" $cagentConfPath -Force
-  If (-not (Test-Path "$cagentConfPath" )) {
-    Write-Host "Caldera Agent configuration failed. Unable to retrieve config from resources folder."
-  }
-  Start-Process -FilePath $cagentPath -ArgumentList '--startup', 'auto', 'install' -Wait
-  Start-Process -FilePath $cagentPath -ArgumentList 'start' -Wait
-} Else {
-  Write-Host "Caldera Agent is already installed. Moving on."
-}
-Start-Sleep 5
-If ((Get-Service -name cagent).Status -ne "Running") {
-  throw "Caldera Agent service not running"
-}
-Write-Host "Cagent installation complete!"

Vagrant/scripts/install-choco-extras.ps1

@@ -10,7 +10,7 @@ If (-not (Test-Path "C:\ProgramData\chocolatey")) {
 }
 
 Write-Host "Installing Chocolatey extras..."
-choco install -y wireshark
+choco install -y --limit-output wireshark
-choco install -y winpcap
+choco install -y --limit-output winpcap
 
 Write-Host "Choco addons complete!"

Vagrant/scripts/install-redteam.ps1

@@ -2,19 +2,7 @@
 
 Write-Host "Installing Red Team Tooling..."
 
-
+# Windows Defender should be disabled already by O&O ShutUp10
-# Disable Windows Defender realtime scanning before downloading Mimikatz and drop the firewall
-If ($env:computername -eq "win10") {
-  If (Test-Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows Defender")
-  {
-    Remove-Item "HKLM:\SOFTWARE\Policies\Microsoft\Windows Defender" -Recurse -Force
-  }
-  gpupdate /force | Out-String
-  Write-Host "Disabling Windows Defender Realtime Monitoring..."
-  Set-MpPreference -ExclusionPath C:\commander.exe, C:\Tools
-  set-MpPreference -DisableRealtimeMonitoring $true
-  Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled False
-}
 
 # Purpose: Downloads and unzips a copy of the latest Mimikatz trunk
 Write-Host "Determining latest release of Mimikatz..."

Vagrant/scripts/install-utilities.ps1

@@ -10,11 +10,11 @@ If (-not (Test-Path "C:\ProgramData\chocolatey")) {
 Write-Host "Installing utilities..."
 If ($(hostname) -eq "win10") {
   # Because the Windows10 start menu sucks
-  choco install -y classic-shell -installArgs ADDLOCAL=ClassicStartMenu
+  choco install -y --limit-output classic-shell -installArgs ADDLOCAL=ClassicStartMenu
-  reg import "c:\vagrant\resources\windows\classic_shell_win7.reg"
+  & "C:\Program Files\Classic Shell\ClassicStartMenu.exe" "-xml" "c:\vagrant\resources\windows\MenuSettings.xml"
 }
-choco install -y NotepadPlusPlus
+choco install -y --limit-output NotepadPlusPlus
-choco install -y GoogleChrome
+choco install -y --limit-output GoogleChrome
-choco install -y WinRar
+choco install -y --limit-output WinRar
 
 Write-Host "Utilties installation complete!"

build.ps1

@@ -338,10 +338,6 @@ function download {
 
 function post_build_checks {
 
-  Write-Host '[post_build_checks] Running Caldera Check.'
-  $CALDERA_CHECK = download -URL 'https://192.168.38.105:8888' -PatternToMatch '<title>CALDERA</title>'
-  Write-Host "[post_build_checks] Cladera Result: $CALDERA_CHECK"
-
   Write-Host '[post_build_checks] Running Splunk Check.'
   $SPLUNK_CHECK = download -URL 'https://192.168.38.105:8000/en-US/account/login?return_to=%2Fen-US%2F' -PatternToMatch 'This browser is not supported by Splunk'
   Write-Host "[post_build_checks] Splunk Result: $SPLUNK_CHECK"
@@ -354,9 +350,6 @@ function post_build_checks {
   $ATA_CHECK = download -URL 'https://192.168.38.103' -SuccessOn401
   Write-Host "[post_build_checks] ATA Result: $ATA_CHECK"
 
-  if ($CALDERA_CHECK -eq $false) {
-    Write-Warning 'Caldera failed post-build tests and may not be functioning correctly.'
-  }
   if ($SPLUNK_CHECK -eq $false) {
     Write-Warning 'Splunk failed post-build tests and may not be functioning correctly.'
   }

build.sh

@@ -261,7 +261,6 @@ vagrant_reload_host() {
 post_build_checks() {
   # If the curl operation fails, we'll just leave the variable equal to 0
   # This is needed to prevent the script from exiting if the curl operation fails
-  CALDERA_CHECK=$(curl -ks -m 2 https://192.168.38.105:8888 | grep -c '302: Found' || echo "")
   SPLUNK_CHECK=$(curl -ks -m 2 https://192.168.38.105:8000/en-US/account/login?return_to=%2Fen-US%2F | grep -c 'This browser is not supported by Splunk' || echo "")
   FLEET_CHECK=$(curl -ks -m 2 https://192.168.38.105:8412 | grep -c 'Kolide Fleet' || echo "")
   ATA_CHECK=$(curl --fail --write-out "%{http_code}" -ks https://192.168.38.103 -m 2)
@@ -271,16 +270,13 @@ post_build_checks() {
   # Associative arrays are only supported in bash 4 and up
   if [ "$BASH_MAJOR_VERSION" -ge 4 ]; then
     declare -A SERVICES
-    SERVICES=(["caldera"]="$CALDERA_CHECK" ["splunk"]="$SPLUNK_CHECK" ["fleet"]="$FLEET_CHECK" ["ms_ata"]="$ATA_CHECK")
+    SERVICES=(["splunk"]="$SPLUNK_CHECK" ["fleet"]="$FLEET_CHECK" ["ms_ata"]="$ATA_CHECK")
     for SERVICE in "${!SERVICES[@]}"; do
       if [ "${SERVICES[$SERVICE]}" -lt 1 ]; then
         (echo >&2 "Warning: $SERVICE failed post-build tests and may not be functioning correctly.")
       fi
     done
   else
-    if [ "$CALDERA_CHECK" -lt 1 ]; then
-      (echo >&2 "Warning: Caldera failed post-build tests and may not be functioning correctly.")
-    fi
     if [ "$SPLUNK_CHECK" -lt 1 ]; then
       (echo >&2 "Warning: Splunk failed post-build tests and may not be functioning correctly.")
     fi

ci/circle_workflows/packer_and_vagrant_changes.sh

@@ -43,8 +43,8 @@ while [ "$MINUTES_PAST" -lt 400 ]; do
   STATUS=$(curl $IP_ADDRESS)
   if [ "$STATUS" == "building" ]; then
     echo "$STATUS"
-    scp -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/|| echo "Vagrant log not yet present"
+    scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/|| echo "Vagrant log not yet present"
-    scp -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/packer_build.log || echo "Packer log not yet present"
+    scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/packer_build.log || echo "Packer log not yet present"
     sleep 300
     ((MINUTES_PAST += 5))
   else
@@ -53,9 +53,9 @@ while [ "$MINUTES_PAST" -lt 400 ]; do
 done
 if [ "$MINUTES_PAST" -gt 400 ]; then
   echo "Serer timed out. Uptime: $MINUTES_PAST minutes."
-  scp -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/ || echo "Vagrant log not yet present"
+  scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/ || echo "Vagrant log not yet present"
-  scp -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/packer_build.log || echo "Packer log not yet present"
+  scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/packer_build.log || echo "Packer log not yet present"
-  curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
+  curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
   exit 1
 fi
 
@@ -63,11 +63,11 @@ fi
 echo $STATUS
 if [ "$STATUS" != "success" ]; then
   echo "Build failed. Cleaning up server with ID $DEVICE_ID"
-  scp -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/ || echo "Vagrant log not yet present"
+  scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/ || echo "Vagrant log not yet present"
-  scp -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/packer_build.log || echo "Packer log not yet present"
+  scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/packer_build.log || echo "Packer log not yet present"
-  curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
+  curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
   exit 1
 fi
 echo "Build was successful. Cleaning up server with ID $DEVICE_ID"
-curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
+curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
 exit 0

ci/circle_workflows/packer_changes.sh

@@ -9,7 +9,7 @@ fi
 
 ## Provision two Type1 baremetal Packet.net servers
 echo "Provisioning packerwindows2016 on Packet.net"
-SERVER1_ID=$(curl -X POST -s --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" -d '{ "facility": "sjc1", "plan": "baremetal_1", "hostname": "packerwindows2016", "description": "testing", "billing_cycle": "hourly", "operating_system": "ubuntu_16_04", "userdata": "", "locked": "false", "project_ssh_keys":["315a9565-d5b1-41b6-913d-fcf022bb89a6", "755b134a-f63c-4fc5-9103-c1b63e65fdfc"] }' 'https://api.packet.net/projects/0b3f4f2e-ff05-41a8-899d-7923f620ca85/devices' | jq ."id" | tr -d '"')
+SERVER1_ID=$(curl -s -X POST -s --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" -d '{ "facility": "sjc1", "plan": "baremetal_1", "hostname": "packerwindows2016", "description": "testing", "billing_cycle": "hourly", "operating_system": "ubuntu_16_04", "userdata": "", "locked": "false", "project_ssh_keys":["315a9565-d5b1-41b6-913d-fcf022bb89a6", "755b134a-f63c-4fc5-9103-c1b63e65fdfc"] }' 'https://api.packet.net/projects/0b3f4f2e-ff05-41a8-899d-7923f620ca85/devices' | jq ."id" | tr -d '"')
 if [ "$(echo -n $SERVER1_ID | wc -c)" -ne 36 ]; then
   echo "Server may have failed provisionining. Device ID is set to: $SERVER1_ID"
   exit 1
@@ -19,7 +19,7 @@ echo "packerwindows2016 successfully provisioned with ID: $SERVER1_ID"
 sleep 5 # Wait a bit before issuing another provision command
 
 echo "Provisioning packerwindows10 on Packet.net"
-SERVER2_ID=$(curl -X POST -s --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" -d '{ "facility": "sjc1", "plan": "baremetal_1", "hostname": "packerwindows10", "description": "testing", "billing_cycle": "hourly", "operating_system": "ubuntu_16_04", "userdata": "", "locked": "false", "project_ssh_keys":["315a9565-d5b1-41b6-913d-fcf022bb89a6", "755b134a-f63c-4fc5-9103-c1b63e65fdfc"] }' 'https://api.packet.net/projects/0b3f4f2e-ff05-41a8-899d-7923f620ca85/devices' | jq ."id" | tr -d '"')
+SERVER2_ID=$(curl -s -X POST -s --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" -d '{ "facility": "sjc1", "plan": "baremetal_1", "hostname": "packerwindows10", "description": "testing", "billing_cycle": "hourly", "operating_system": "ubuntu_16_04", "userdata": "", "locked": "false", "project_ssh_keys":["315a9565-d5b1-41b6-913d-fcf022bb89a6", "755b134a-f63c-4fc5-9103-c1b63e65fdfc"] }' 'https://api.packet.net/projects/0b3f4f2e-ff05-41a8-899d-7923f620ca85/devices' | jq ."id" | tr -d '"')
 if [ "$(echo -n $SERVER2_ID | wc -c)" -ne 36 ]; then
   echo "Server may have failed provisionining. Device ID is set to: $SERVER2_ID"
   exit 1
@@ -32,8 +32,8 @@ echo "Sleeping 5 more minutes (CircleCI Keepalive)"
 sleep 300
 
 ## Recording the IP address of the newly provisioned Packet servers
-SERVER1_IP_ADDRESS=$(curl -X GET --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" "https://api.packet.net/devices/$SERVER1_ID/ips" | jq ."ip_addresses[0].address" | tr -d '"')
+SERVER1_IP_ADDRESS=$(curl -s -X GET --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" "https://api.packet.net/devices/$SERVER1_ID/ips" | jq ."ip_addresses[0].address" | tr -d '"')
-SERVER2_IP_ADDRESS=$(curl -X GET --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" "https://api.packet.net/devices/$SERVER2_ID/ips" | jq ."ip_addresses[0].address" | tr -d '"')
+SERVER2_IP_ADDRESS=$(curl -s -X GET --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" "https://api.packet.net/devices/$SERVER2_ID/ips" | jq ."ip_addresses[0].address" | tr -d '"')
 
 # Copy repo to Packet servers
 # TODO: Tar up the repo and expand it remotely
@@ -54,8 +54,8 @@ while [ "$MINUTES_PAST" -lt 150 ]; do
   SERVER2_STATUS=$(curl $SERVER2_IP_ADDRESS)
   if [[ "$SERVER1_STATUS" == "building" ]] || [[ "$SERVER2_STATUS" == "building" ]]; then
     echo "$SERVER1_STATUS" :: "$SERVER2_STATUS"
-    scp -i ~/.ssh/id_rsa root@"$SERVER1_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server1_packer.log
+    scp -q -i ~/.ssh/id_rsa root@"$SERVER1_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server1_packer.log
-    scp -i ~/.ssh/id_rsa root@"$SERVER2_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server2_packer.log
+    scp -q -i ~/.ssh/id_rsa root@"$SERVER2_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server2_packer.log
     sleep 300
     ((MINUTES_PAST += 5))
   fi
@@ -64,8 +64,8 @@ while [ "$MINUTES_PAST" -lt 150 ]; do
   fi
   if [ "$MINUTES_PAST" -gt 150 ]; then
     echo "Serer timed out. Uptime: $MINUTES_PAST minutes."
-    curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER1_ID"
+    curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER1_ID"
-    curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER2_ID"
+    curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER2_ID"
     exit 1
   fi
 done
@@ -75,17 +75,17 @@ echo "Server1 Status: $SERVER1_STATUS"
 echo "Server2 Status: $SERVER2_STATUS"
 if [ "$SERVER1_STATUS" != "success" ]; then
   echo "Build failed. Cleaning up server with ID $SERVER1_ID"
-  scp -i ~/.ssh/id_rsa root@"$SERVER1_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server1_packer.log || echo "Serveer1 packer_build.log not available yet"
+  scp -q -i ~/.ssh/id_rsa root@"$SERVER1_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server1_packer.log || echo "Serveer1 packer_build.log not available yet"
-  curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER1_ID"
+  curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER1_ID"
   exit 1
 fi
 if [ "$SERVER2_STATUS" != "success" ]; then
   echo "Build failed. Cleaning up server with ID $SERVER2_ID"
-  scp -i ~/.ssh/id_rsa root@"$SERVER2_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server2_packer.log || echo "Server2 packer_build.log not available yet"
+  scp -q -i ~/.ssh/id_rsa root@"$SERVER2_IP_ADDRESS":/opt/DetectionLab/Packer/packer_build.log /tmp/artifacts/server2_packer.log || echo "Server2 packer_build.log not available yet"
-  curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER2_ID"
+  curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER2_ID"
   exit 1
 fi
 echo "Builds were successful. Cleaning up servers with IDs $SERVER1_ID and $SERVER2_ID"
-curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER1_ID"
+curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER1_ID"
-curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER2_ID"
+curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$SERVER2_ID"
 exit 0

ci/circle_workflows/vagrant_changes.sh

@@ -56,7 +56,7 @@ while [ "$MINUTES_PAST" -lt 180 ]; do
   if [ "$MINUTES_PAST" -gt 180 ]; then
     echo "Serer timed out. Uptime: $MINUTES_PAST minutes."
     scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/
-    curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
+    curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
     exit 1
   fi
 done
@@ -66,9 +66,9 @@ echo $STATUS
 if [ "$STATUS" != "success" ]; then
   scp -q -i ~/.ssh/id_rsa root@"$IP_ADDRESS":/opt/DetectionLab/Vagrant/vagrant_up_*.log /tmp/artifacts/
   echo "Build failed. Cleaning up server with ID $DEVICE_ID"
-  curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
+  curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
   exit 1
 fi
 echo "Build was successful. Cleaning up server with ID $DEVICE_ID"
-curl -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
+curl -s -X DELETE --header 'Accept: application/json' --header 'X-Auth-Token: '"$PACKET_API_TOKEN" 'https://api.packet.net/devices/'"$DEVICE_ID"
 exit 0

ci/manual_machine_bootstrap_vmware.sh

@@ -0,0 +1,65 @@
+#! /bin/bash
+
+# This script is used to manually prepare an Ubuntu 16.04 server for DetectionLab building
+
+SERIALNUMBER="TODO"
+LICENSEFILE="TODO"
+
+sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/g' /etc/apt/sources.list
+
+if [[ "$VAGRANT_ONLY" -eq 1 ]] && [[ "$PACKER_ONLY" -eq 1 ]]; then
+  echo "Somehow this build is configured as both packer-only and vagrant-only. This means something has gone horribly wrong."
+  exit 1
+fi
+
+# Install VMWare Workstation 15
+apt-get update
+apt-get install -y linux-headers-"$(uname -r)" build-essential unzip git ufw apache2 python-pip
+pip install awscli --upgrade --user
+export PATH=$PATH:/root/.local/bin
+
+wget -O VMware-Workstation-Full-15.0.4-12990004.x86_64.bundle "https://download3.vmware.com/software/wkst/file/VMware-Workstation-Full-15.0.4-12990004.x86_64.bundle?HashKey=6f83753e4d9e94da7f920c32b5808033&params=%7B%22custnumber%22%3A%22KipkcHRoJWVlZA%3D%3D%22%2C%22sourcefilesize%22%3A%22472.70+MB%22%2C%22dlgcode%22%3A%22WKST-1504-LX%22%2C%22languagecode%22%3A%22en%22%2C%22source%22%3A%22DOWNLOADS%22%2C%22downloadtype%22%3A%22manual%22%2C%22eula%22%3A%22Y%22%2C%22downloaduuid%22%3A%225caee685-d5ad-4f6b-94db-2ddc4f7f3a97%22%2C%22purchased%22%3A%22N%22%2C%22dlgtype%22%3A%22Product+Binaries%22%2C%22productversion%22%3A%2215.0.4%22%2C%22productfamily%22%3A%22VMware+Workstation+Pro%22%7D&AuthKey=1556427011_a994b5252f29429710c077c8dcab1c19"
+chmod +x VMware-Workstation-Full-15.0.4-12990004.x86_64.bundle
+sudo sh VMware-Workstation-Full-15.0.4-12990004.x86_64.bundle --console --required --eulas-agreed --set-setting vmware-workstation serialNumber $SERIALNUMBER
+
+echo "building" > /var/www/html/index.html
+
+# Set up firewall
+ufw allow ssh
+ufw allow http
+ufw default allow outgoing
+ufw --force enable
+
+git clone https://github.com/clong/DetectionLab.git /opt/DetectionLab
+
+# Install Vagrant
+mkdir /opt/vagrant
+cd /opt/vagrant || exit 1
+wget --progress=bar:force https://releases.hashicorp.com/vagrant/2.2.4/vagrant_2.2.4_x86_64.deb
+dpkg -i vagrant_2.2.4_x86_64.deb
+vagrant plugin install vagrant-reload
+vagrant plugin install vagrant-vmware-desktop
+vagrant plugin license vagrant-vmware-desktop $LICENSEFILE
+wget --progress=bar:force "https://releases.hashicorp.com/vagrant-vmware-utility/1.0.7/vagrant-vmware-utility_1.0.7_x86_64.deb"
+dpkg -i vagrant-vmware-utility_1.0.7_x86_64.deb
+
+# Make the Vagrant instances headless
+cd /opt/DetectionLab/Vagrant || exit 1
+sed -i 's/v.gui = true/v.gui = false/g' Vagrantfile
+
+# Install Packer
+mkdir /opt/packer
+cd /opt/packer || exit 1
+wget --progress=bar:force https://releases.hashicorp.com/packer/1.3.2/packer_1.3.2_linux_amd64.zip
+unzip packer_1.3.2_linux_amd64.zip
+cp packer /usr/local/bin/packer
+
+# Make the Packer images headless
+cd /opt/DetectionLab/Packer || exit 1
+for file in *.json; do
+  sed -i 's/"headless": false,/"headless": true,/g' "$file";
+done
+
+# Ensure the script is executable
+chmod +x /opt/DetectionLab/build.sh
+cd /opt/DetectionLab || exit 1