Honeypot maintenance

STINGARv2 Honeypots run on virtual machines and collect data from attackers. The honeypots immediately send the attack data back to their STINGAR server but also retain a local copy of the attack logs inside the honeypot's container running on the virtual machine (vm). Over time (typically many months) these log files can fill all the available storage space on the vm and cause the honeypot container to run out of resources and potentially stop working.

To avoid this it is important to regularly check your honeypot deployments to ensure their logs are cleaned out, i.e. deleted (or archived). Typically this clean up can be performed quarterly (sometimes annually) but if the honeypot is exposed directly to the external internet and has a large number of attacks it may need to be cleaned up more regularly (i.e. monthly).

Manually cleaning up honeypot log files periodically

The simplest way to keep your honeypots running is to manually login to the honeypot vm, then connect to the container by executing a shell from docker-compose :

Steps to connect to honeypot container on vm

First, remotely connect to your honeypot vm (using ssh)

ssh -i ~/.ssh/my_honeypot_keys.pem myusername@123.456.78.9

Once you are logged into the honeypot vm, ensure you are in the correct directory where you will find the honeypot code. Usually this will be the home directory of the 'stingar' user,

cd /home/stingar

where you should find 2 files : docker-compose.yml and stingar-hp.env. From this directory, run this command from the honeypot vm shell:

For a Cowrie Honeypot

% docker-compose exec cowrie /bin/sh

This connects the terminal to a shell (sh) running inside the honeypot's docker container - this contains the source code of the honeypot application. Once inside the container, navigate to the honeypot application's log directory and delete the log files:

<running sh inside the cowrie container>

$ cd /opt/cowrie/var/log/cowrie
$ ls -l
total 2968
-rw-rw-r--. 1 cowrie cowrie 1033176 Jan 16 15:08 cowrie.json
-rw-rw-r--. 1 cowrie cowrie  505232 Jan 15 23:59 cowrie.json.2025-01-15
-rw-r--r--. 1 cowrie cowrie  939741 Jan 16 15:08 cowrie.log
-rw-r--r--. 1 cowrie cowrie  487856 Jan 15 23:59 cowrie.log.2025-01-15
$ rm cowrie.json*
$ rm cowrie.log*
$ ls -l
total 0
$ exit

For a Dionaea Honeypot

% docker-compose exec dionaea /bin/bash

Once inside the container, navigate to the honeypot application's log directory and delete the log files:

<running bash inside the dionaea container>

root@73acbf5dd414:/opt/dionaea# 
root@73acbf5dd414:/opt/dionaea# cd /opt/dionaea/var/dionaea
root@73acbf5dd414:/opt/dionaea/var/dionaea# ls -l
total 227572
drwxrwxr-x 1 nobody nogroup      4096 Jun 30  2022 binaries
drwxrwxr-x 4 nobody nogroup      4096 Jan 16 00:08 bistreams
-rw-r--r-- 1 root   root      1441792 Jan 16 17:08 dionaea-errors.log
-rw-r--r-- 1 root   root     21211783 Jan 15 17:08 dionaea-errors.log.gz
-rw-r--r-- 1 root   root    228286464 Jan 16 17:08 dionaea.log
-rw-r--r-- 1 root   root   1427282264 Jan 15 17:08 dionaea.log.gz
-rw-r--r-- 1 nobody nogroup   3248128 Jan 16 17:08 dionaea.sqlite
-rw-r--r-- 1 nobody nogroup  13284223 Jan 15 17:08 dionaea.sqlite.gz
drwxrwxr-x 1 nobody nogroup      4096 Jun 30  2022 roots
drwxrwxr-x 1 nobody nogroup      4096 Jun 30  2022 share

root@73acbf5dd414:/opt/dionaea/var/dionaea# rm *.gz
root@73acbf5dd414:/opt/dionaea/var/dionaea# rm -r bistreams/*
root@73acbf5dd414:/opt/dionaea/var/dionaea# exit

Confirming free space on vm

You can confirm the amount of free storage space with the linux commands from the vm to ensure there is available space to continue using the honeypot. If not, then there may be other applications or system log files filling up the honeypot's vm shared storage. Some useful linux commands to discover where the problem lies are:

To find large files on vm storage

% df -Th

OR find specific files larger than a certain size (e.g. 1Gbyte)

% find / -size +1G

OR list sub directories with total

du .

Automatic log cleanup with host cron job

A more convenient option is to set up a timed script on the honeypot vm using linux crontab tool to regularly clean up the honeypot log files.

This requires a 1-time setup of the crontab schedule and creating a script file but will then automatically clean up the honeypot regularly without any manual intervention allowing the honeypot to run for many months, even years, without intervention.

Steps required for automatic setup

1. Create a shell script file

You will need to copy this script to the same base directory as the other STINGAR honeypot files docker-compose.yml & stingar-hp.env which is typically under the vm's stingar user directory /home/stingar (your vm may have a different user or home directory, ensure this directory is referenced in the crontab command below)

Example cleanup script for cowrie honeypot cowrie_log_cleanup.sh

#!/bin/bash

# Check the container's Cowrie log directory exists
directory="/opt/cowrie/var/log/cowrie"

if ! docker-compose exec cowrie [ -d "$directory" ]; then
  echo "Error: Directory '$directory' does not exist in the container 'cowrie'."
  exit 1
fi

# Match and delete log files in the directory
docker-compose exec cowrie find "$directory" -type f -name "*.log*" -exec rm -v {} \;
docker-compose exec cowrie find "$directory" -type f -name "*.json*" -exec rm -v {} \;

current_time=$(date '+%Y-%m-%d %H:%M:%S')
echo "'$current_time' : Log Cleanup : all log & json files and download sub-directories have been deleted from '$directory' in the container 'cowrie'."

Example cleanup script for Dionaea honeypot dionaea_log_cleanup.sh

# Assign the Dionaea log directory 
directory="/opt/dionaea/var/dionaea"

# Check if the directory exists
if ! docker-compose exec dionaea [ -d "$directory" ]; then
  echo "Error: Log Directory '$directory' does not exist in the container 'dionaea'."
  exit 1
fi

# Find and delete log files in the directory
# Find and delete all .gz files in the directory inside the container
docker-compose exec dionaea find "$directory" -type f -name "*.gz" -exec rm -v {} \;

# Find and remove all log sub-directories within "bistreams", but not "bistreams" itself, inside the container
if docker-compose exec dionaea [ -d "$directory/bistreams" ]; then
  docker-compose exec dionaea find "$directory/bistreams" -mindepth 1 -type d -exec rm -rv {} \;
fi

current_time=$(date '+%Y-%m-%d %H:%M:%S')
echo "'$current_time' : Log cleanup : all log files and bistreams sub-directories have been deleted from '$directory' in the container 'dionaea'.

1.1 Create a new script file in the honeypot's home directory

Create a new file (i.e. cowrie_log_cleanup.sh or dionaea_log_cleanup.sh) with your editor and copy & paste the above script code into your editor then save (or copy) the file into the correct directory. cp cowrie_log_cleanup.sh /home/stingar

Ensure the file it has execute permissions set chmod 755 cowrie_log_cleanup.sh

1.2 Optionally create an empty Log file to capture the cleanup event

The cleanup script will delete the honeypot log files and output a 1 line message on this VM with a record of when the log files were deleted. rather than output this message to the stdout console, you may optionally choose to capture this output in a log file for future reference. You will first need to create an empty file and set its write permissions appropriately.

touch /var/log/STINGAR_cleanup.log`
chmod 664 /var/log/STINGAR_cleanup.log

2. Set the crontab schedule when to execute the script

To setup a schedule to execute the cleanup script to delete the honeypot logs (and optionally capture the confimation message in /var/log/STINGAR_cleanup.log file, use the linux crontab command.

% crontab -e

The crontab tool opens the default editor with a new cron file. Edit this file with a single line (see below) to schedule the cleanup script task.

To execute a cowrie cleanup script at 2:00am on first day each month:

0 2 1 * * /home/stingar/cowrie_log_cleanup.sh >> /var/log/STINGAR_cleanup.log 2>&1

To execute on a weekly schedule (on first day of each week (Sundays) at 2:00am) you can use:

0 2 * * 0 /home/stingar/cowrie_log_cleanup.sh >> /var/log/STINGAR_cleanup.log 2>&1

Once you save this file and exit the editor the cronjob is created and will execute according the the schedule defined. To confirm this use % crontab -l to list all cron events scheduled on this vm. You can update the schedule anytime using the same % crontab -e command.

Once this cronjob is scheduled, it will run indefinitely (even after vm reboots). You may periodically check the /var/log/STINGAR_cleanup.log on the vm to ensure the files are being deleted by running % docker-compose exec cowrie /bin/sh cat /var/log/STINGAR_cleanup.log