What a weird time to be online. Haven’t updated this in a while, and everything else has kind of fallen apart too, right? anyways lets ramble
HOUSEKEEPING: yes, I have continued to do the Holiday Hack. No one saw fit to give me an honorable mention this previous year, probably because I complained too much about AI and other errors (and I suppose, in light of my reaction to those errors and my own burnout, did not adhere to my typical quality standards. This is okay and is necessary.) They’re not worth detailing, but the change of scenery is still a welcome December distraction. We’re still tracking them on GitHub, and that’s all I’ll report:
And now to the real ramble about a home Splunk environment
Accidentally used some time off of work today to get involved with my home setup, and it all started because some endpoint one of my wife’s games uses was getting blocked by my pihole. “Oh, I have those logs in Splunk, let me figure out what’s been breaking.” of course it would not be so simple
If you’ve ever used Splunk before, you might be familiar with the “Getting Data In” acronym, GDI. I feel any time I have to work with “Getting Data In,” GDI feels appropriate because I whine “god damn it” the whole time. Turns out that when I started this “project” months ago it was incomplete and something had failed so I had no pihole logs, just the normal system stuff still working over syslog because I couldn’t get the Splunk Universal Forwarder running on the pihole correctly at the time, and at SOME POINT I had managed to successfully send over a set of logs briefly. I did use a guide (https://satiex.net/infosec/guides/forward-pi-hole-logs-to-splunk/), but my experience was not 1:1 with the article and I’m not offering notes here. Not super proud of the things I did to make it function instead of doing the syslog route, but my minimum viable product is still probably depressingly better than what other people are absolutely running in production so we gon’ roll with it. If I could tell you the things I’ve seen neither of us would leave this blog feeling Okay. ANYWAYS read more if you feel like it.
Universal Forwarder for Windows and Sysmon
After resolving that, it seemed silly not to consider what other sources I could be sending to Splunk. Professionally and personally, I manage and interact with a lot of Splunk infrastructure, so all practice is good practice. So, next steps were to configure sysmon (used this repo: https://github.com/ion-storm/sysmon-config) and the universal forwarder on my Windows hosts.
Initially, I thought I’d be smart and add the sysmon event log during the initial config. The selection occurs before UAC elevation, so the installer complains that you don’t have access to the file because it requires admin. I tried to go around this by modifying the file permissions temporarily–but don’t bother. Just add it to your inputs.conf after the fact (I also used some context from the existing entries at C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf). Wait! That wasn’t enough. 🙃
Access denied. I tried adding the NT Service\SplunkForwarder account to the ACL on the .evtx file which did not help. Ultimately changing the service from the virtual user to just Local Service did work, and I’m not getting paid to figure out what the catch is. We’re rolling with it, remember? After changing the service account and restarting the service, sysmon logs are up and happy. Unfortunately because I wasn’t verifying my work as I went, I had to go fix this on all my Windows forwarders. Whomp whomp. Is it possible that had I done the local account with the initial ACL changes, would it have worked? MAYBE! But I kind of doubt it, because it wasn’t in my inputs.conf already so it may have been tracked somewhere else…that wasn’t as functional.
Unfortunately, my home network is very consumer and I can’t get any good logs from that–but the pihole is providing DHCP and DNS info–close enough. What else can we throw at it?
External Server Logs
Here’s a dilemma: this server and other functions, I run out of a Vultr VPS. I’d like to get the logs from this server down to my Splunk Enterprise environment. But my Splunk Enterprise server is a Rocky Linux install running on Hyper-V in my private home network, and my VPS is ~elsewhere~. While it is technically possible to do some port forwarding to my internal host, my home infra doesn’t allow me to scope that, and I wasn’t looking into how to restrict access to only approved Splunk Forwarders, nor was I comfortable with the idea of exposing any ports related to the service externally if I couldn’t properly constrain it.
My solution was to create a limited user account on the VPS and grant that account read access to specific logs and locations. From my internal Splunk server, I generated an SSH keypair and added the public key to the authorized_hosts on the limited user on my server.
I configured a cron job on the internal Splunk server to copy specific logs from the remote server to a local directory on a specific schedule using rsync (after verifying that the commands worked manually, of course).
Then, from Splunk Enterprise, I just added a new File & Directory data input for the folder that the rsync results were going to. If I were thinking like a SIEM this is way different data and way different context, but this is just practice and giving me interesting and easier access to data. In defining the input, I was able to manually specify the correct host value, rather than the locally monitored entries appearing as if they were from the Splunk server.
Conclusion, right?
Okay! Now I just need to work on making sure that I have all the right Splunk apps I need to assist in field extraction. I was going to do Entra ID stuff but I ran out of steam.😿
The enterprising reader will note that while I have spent a large portion of my day GDI I have not, not even a little, determined what entries may need to be exempted from my pihole config to let my wife play a mobile game. whoops. Maybe, just maybe, I’ll be more prepared this evening to try again.
Anways thanks y’all for sticking around. Other interesting projects have been handling the Google Domains DEBACLE in transferring domains around and testing out Cloudflare (yes, I know the controversy; no, i don’t sanction it; and ultimately, experience is the best teacher and it was time to learn more about how many modern sites function). Outside of that I’m trying to learn how to be a functioning human being again though therapy, because people are harder to deal with than devices, and that is having mixed results. Big “despite everything, it’s still you” energy, yet somehow, the alternatives are even worse! Thanks for reading, if you’re a spam bot commenting on this or any other post, fuck you and everyone else associated with you.