{"id":1100,"date":"2014-09-18T20:00:55","date_gmt":"2014-09-19T01:00:55","guid":{"rendered":"http:\/\/www.nathanhunstad.com\/blog\/?p=1100"},"modified":"2014-09-18T20:09:02","modified_gmt":"2014-09-19T01:09:02","slug":"log-file-automation","status":"publish","type":"post","link":"http:\/\/www.nathanhunstad.com\/blog\/2014\/09\/log-file-automation\/","title":{"rendered":"Log File Automation"},"content":{"rendered":"<p>When <a href=\"http:\/\/www.nathanhunstad.com\/blog\/2014\/08\/splunk-reporting-mapping-brute-force-attempts\/\" target=\"_blank\">I set up Splunk reporting<\/a> for my website, it was a purely manual process, and I left for the future the goal of pulling the logs automatically. Since then, that\u2019s exactly what I\u2019ve done, so now it runs completely automatically. Below is how.<\/p>\n<p><!--more--><\/p>\n<p>The first thing I needed to do was enable create SSH keys so that I could log into my website without any user intervention. Following <a href=\"http:\/\/support.hostgator.com\/articles\/specialized-help\/technical\/ssh-keying-through-putty-on-windows-or-linux\" target=\"_blank\">this how-to<\/a> from HostGator, I used PuTTYGen to create the key pair, then I placed the public key on my website. I could then SSH over port 2222 (instead of port 22) to my website, after a change to my firewall to allow communication over port 2222. To automate the process on my server, I created a <span style=\"font-family: 'Courier New';\">config<\/span> file in the <span style=\"font-family: 'Courier New';\">~\/.ssh <\/span>directory so that SSH would automatically use the right private key file when connecting. Step one was done.<\/p>\n<p>Step two was to actually grab the files. For this, I finally settled on using <span style=\"font-family: 'Courier New';\">rsync<\/span>, a very handy *nix file-syncing utility. I wrote a bash script to rsync over SSH to pull down both the <span style=\"font-family: 'Courier New';\">error.php<\/span> log, as well as the Apache logs for my website (for the latter, I configured cPanel to create a ZIP file every day containing the day\u2019s access logs). Some <span style=\"font-family: 'Courier New';\">sed<\/span> work, file renaming, and moving then happens, and the end result is that those files are dumped in a directory that Splunk is monitoring. Splunk then sees the new files and indexes them appropriately. Voil\u00e0!<\/p>\n<p>I set up a <span style=\"font-family: 'Courier New';\">cron<\/span> job to run this as the <span style=\"font-family: 'Courier New';\">splunk<\/span> user once a day. It\u2019s not real time, but it\u2019s as close as you can get with the limitations around that Apache log. If, for some reason, the daily job doesn\u2019t run correctly, there\u2019s actually no problem: the next run will get the data with no data loss, since both the access logs and the <span style=\"font-family: 'Courier New';\">error.php<\/span> file will simply keep logging I grab them.<\/p>\n<p>Next steps are to tweak this and see if I can grab failed logins for my blog and photo site as well; that way, I could correlate across all three login portals to see if people are trying to access just one part of the site or multiple. Unfortunately, neither WordPress nor ZenPhoto log failed authentication attempts by default, but there are solutions out there.<\/p>\n<p>Now that I have some data, I\u2019ll have to play around with some more dashboards for my website data in Splunk and update that. More to come!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When I set up Splunk reporting for my website, it was a purely manual process, and I left for the future the goal of pulling the logs automatically. Since then, that\u2019s exactly what I\u2019ve done, so now it runs completely automatically. Below is how.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[19],"tags":[252],"class_list":["post-1100","post","type-post","status-publish","format-standard","hentry","category-tech-2","tag-splunk","entry"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/posts\/1100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/comments?post=1100"}],"version-history":[{"count":2,"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/posts\/1100\/revisions"}],"predecessor-version":[{"id":1105,"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/posts\/1100\/revisions\/1105"}],"wp:attachment":[{"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/media?parent=1100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/categories?post=1100"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.nathanhunstad.com\/blog\/wp-json\/wp\/v2\/tags?post=1100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}