Defeating Slow Distributed SSH Brute Force Attacks with Fail2Ban

January 16, 2017 Tech No comments , , ,

If you are like me and you run a public-facing SSH listener you will undoubtedly start to see a ton of failed SSH login attempts in /var/log/auth.log.  If you are a paranoid cyber security professional like myself, then this really irks you.  While your passwords are longer and more complex than 99% of those used, you still know that there is a tiny, slim-to-none chance that one of those login attempts might actually work.

Most of these failed SSH logins come from botnets that are looking for low-hanging fruit in the form of poorly secured Linux servers.  When these botnets find one, they’ll log in, attempt to gain a foothold through any type of vulnerability and spread their botnet putridness.  Luckily, security wary sysadmins have a lot of wonderful tools available to them.  The one that I love to go to is Fail2Ban.

Fail2Ban enters the arena

Fail2Ban is a free utility that scans all sorts of logs looking for indicators of various types of malicious activity – multiple failed logins, attempts to leverage exploits, etc.  When it detects a malicious attempt it temporarily adds the offender to the local system’s firewall for a set period of time.

Just by installing it ( sudo apt-get update && sudo apt-get install fail2ban ), the utility will immediately begin defending against the exact type of attempt to brute force a passworded SSH listener that I write about tonight.

It works great for most situations, however a lot of bot herders are well aware of Fail2Ban and have configured their bots to avoid detection by it by throttling back their attempts.  Most bots don’t care and will continue to attempt brute forcing even with a drop rule on the target server’s firewall.  But there are smart ones that realize that the default configuration of fail2ban puts a 10 minute ban on any IP that makes 5 failed login attempts over a 10 minute period.  You’ll see some of the offenders make 4 attempts and wait 10 minutes or space out their attempts so that in any given 10 minute period, there are only 4 failed attempts.  Combine this with 20 or more different IP addresses all doing the same thing and the number of password guesses against your SSH listener adds up pretty quickly.  Pretty sneaky.

I don’t know about you, but I wear a tinfoil hat on my head when it comes to security of my systems.  Luckily Fail2Ban has a juicy configuration file to dive into.  Here’s what I did to drastically destroy chances of my servers being successfully brute forced:

Using my favorite lazy editor, I edited /etc/fail2ban/jail.conf.  The parameters I changed and their values are below:

  • bantime = 31536000
  • findtime = 84600

Those values are in seconds.  Yes, offenders are banned for an entire year.  Yes, fail2ban now looks for 5 login failures in a 24 hour period.  As today’s youth might say:  Savage.

You can also whitelist IP addresses by adding whitelisted addresses, networks, or CDIR notations to the following line:

ignoreip =

If you are following along, don’t forget to restart Fail2Ban:

sudo service fail2ban restart

The theory behind this is two-fold:

  1. I don’t use a password to log into my SSH listeners so any attempts to authenticate with a password is not a legitimate login attempt.  I use key-based authentication and you should too.  DigitalOcean has a great guide on configuring key-based authentication for SSH..  If you don’t want to use key-based authentication, another option is two-factor authentication.  DigitalOcean also has an outstanding and easy to follow guide on how to set it up using Google’s two-factor authentication platform.
  2. My hopes are that within a year the system owner will realize that their system is participating in a botnet and take action to remedy the situation.  Additionally, a year of dropped packets should be enough to make that particular bot report to higher that my IP address is no longer responding.

The Aftermath

When making changes like this, Fail2Ban actually retroactively reviews the existing /var/log/auth.log file to look for offenders.  If you don’t use two-factor authentication yet and you aren’t 100% certain that you haven’t violated the new, stricter rule, you might want to go start fresh and delete that file.  If there is a chance that you failed to authenticate 5 times in a single day, then you may be facing a 1 year drop rule from your server, which could be problematic if SSH is your only way of accessing a remote server.

Defeating Slow Distributed SSH Brute Force Attacks with Fail2Ban

Output of /var/log/fail2ban.log after the changes were made and applied.

I noticed an immediate drop in failed SSH login attempts.  Kibana in my ELK Stack gave me a nice visualization of the drop off.  Before the change to Fail2Ban’s configuration I was seeing anywhere from 8 to 20 failed SSH login attempts every 10 minutes.  Afterwards I am seeing 0 to 4.  The number of IP addresses that were retroactively banned is huge and nearly doubled the size of my sudo iptables -L output.

Defeating Slow Distributed SSH Brute Force Attacks with Fail2Ban

Kibana reports a huge drop off in failed SSH logins.

24 hours later…

Below is the output from Kibana on the server covering the 24 hours leading up to and 24 hours after I put the policy into place.


How to Stop a Begging Dog

January 9, 2017 Video No comments , , ,

Simple File to Max Out your CPU

January 9, 2017 Guides, Tech No comments , , , , , ,

Whether you want to stress test your computer’s CPU to test its stability or maybe the CPU cooler or you are like me and work in an office that doesn’t have a heater that anyone would consider satisfactory (49 degrees, seriously?), running your computer’s CPU at max is actually pretty simple.

You don’t need a special program if you just want to max your CPU, all you need is a simple batch file.

Copy the following text and open notepad and paste the text in.  Then save it somewhere and name it “something.bat” – you can obviously replace the something portion with whatever you like.  Double click it and a black window will open.  Double click it once for each core that your computer has.  See below for a picture showing you how to figure out how many cores your computer has.  Windows will put the entire load of the batch file onto a single core – so you have to run it once for each core.  To shut it down, just close the window.

How it works

It’s very simple and very safe.  You see all the code that is running.  It just creates an infinite loop.

The first line “@echo off” tells it to not output anything to the window.  The second line establishes a label for the program to return to if called upon to do so.  This is a component in a “GoTo loop“.  The third line tells the program to go to loop, which is the second line.  As you can easily see, this program puts the computer into an endless loop…. more commonly referred to as an “infinite loop”.  It will not stop unless acted upon externally such as by closing the window or pressing Control+C.

What does it do to my computer?

Each copy of this batch file that is run will increase the utilization of a single CPU core to nearly 100%.

Your CPU will start to heat up.

Your computer may start to run slower than normal, but this probably will not be the case as windows will prioritize other programs before this gets any time using the CPU.

Batch file code:

@echo off
goto loop


This is what the script looks like when it is running. Closing this window ends the script!

This is Windows Task Manager’s Performance tab. To open this window and see how many cores you have, right click on an empty area of your task bar and click “Start Task Manager”. The boxes I’ve marked in red show how many cores your computer has. You must run the script once for each core. Windows will automatically distribute the load onto the cores. Don’t run it more than the number of cores you have – you won’t have any added effect!

A Day At The Park

January 4, 2017 Family No comments ,

Stepping Off on the Wrong Foot

May 18, 2016 Army Stories No comments

I’ve served with quite a few fuck ups in my career, but this one takes the cake.  I honestly don’t know how this guy got through MEPS.

For PT one morning, my platoon conducted a short 6 mile ruck march.  We started early so we could be done by 0730.  It was dark when we started.

We finish the ruck march and I’m standing around with the Soldiers in my squad.  The dumb one comes up to me and we have this exchange:

Private:  Sarge, my feet hurt.

Me:  We just finished a 6 mile ruck march and we haven’t done one in a while so that’s probably why.

Private:  No, sarge.  They really hurt.  Like more than usual.  Can I take my boots off?

Me:  You have a few minutes.  Change your socks while you are at it.

At this point, the sun is starting to come up so we have some light to see things more clearly.  He sits down and I watch him take his first boot off.  I do a double-take because I thought I saw him take his left boot off of his right foot.  I inquire:

Me:  Hey, did you just take that boot off?

Private:  Roger, Sergeant!

Me:  And you haven’t taken the other boot off since you started the ruck march?

Private:  Roger, Sergeant!

Me:  I want you to look VERY closely as the boot that is on your left foot.  Does that look like your right boot?

Private:  No, Sergeant!

Me:  I think we found why your feet hurt.

How do people like this make it to adulthood?

Golden Showers

May 15, 2016 Army Stories No comments

In Iraq, especially in the summer, it can get hot.  It feels like a hair dryer is blowing in your face nonstop at times.

We came off this mission in the middle of the summer.  Everyone was soaked to the bone in sweat.  Once we parked the trucks in the base, most of the Soldiers got out and started taking off their 65+ pounds of equipment to get some of the hot breeze to hopefully dry some of that sweat and cool them off.

The gunner in my truck starts tossing water bottles from the cooler to the guys so they can rehydrate and cool off.  Some of the guys pour the water bottles on their face for a quick cool rinse.  He also starts tossing any trash out so we can clean the truck out.

One of our guys pours a water bottle on his face and instead of looking refreshed, he tenses up and looks disgusting.

Him:  What the fuck!  That was piss!  I just poured piss all over myself!

I don’t know for sure if he picked a random bottle off the ground or the gunner tossed him a piss bottle on purpose to make it seem like it was a water bottle.  I want to believe it was the latter.

The Infantry Isn’t What He Thought

May 14, 2016 Army Stories No comments ,

The Infantry Isn't What He Thought

The Infantry is known for being home to some fairly unintelligent individuals.  Some are just outright dumb, stupid, or otherwise unintelligent… others just don’t know how to ask questions.  You can’t tell me your recruiter lied to you if you didn’t ask the necessary questions about what you’d be doing in the infantry.

This Soldier didn’t appear happy with his job within the Infantry.  He was a relatively new private and he didn’t seem motivated.  He didn’t seem like he wanted to do anything the Infantry does, to include firing weapons from our impressive arsenal.  So the NCOs of my platoon had him come to the office and we started inquiring as to why he didn’t seem motivated.

NCO:  Why don’t you seem like you want to do anything we do here in the Infantry?

Soldier:  This isn’t what I thought it would be like when I joined the Army.  It’s a completely different job than what I had in mind.

NCO:  What did you think the Infantry did?

Soldier:  I thought the infantry was about taking care of infants.

Stick a fork in me… I’m done!

Simple Math Stumps Simple Private

May 14, 2016 Army Stories No comments ,

Simple Math Stumps Simple Private

Like yesterday’s post, we hit on the fact that the Infantry doesn’t necessarily attract the most intelligent people.

As NCOs we had to keep our Soldiers on their feet and always thinking… always pushing them to be better in every way possible.  Get Smarter and Stronger Everday was a motto some of us embodied.

Physical Training in the morning was the time for Soldiers to push themselves and improve themselves physically.  Throughout the day the NCOs would ask Soldiers questions about their job and life in general to push the envelope of their knowledge.

One of our Soldiers didn’t seem to grasp some common sense concepts and we realized he didn’t truly understand simple things that the average adult should understand.  Here is one such simple question that stumped him:

NCO:  If you drove at 80 Miles Per Hour for an hour, how far would you go?

Soldier:  Uh… I don’t know.

NCO:  Think about it.  80… Miles… Per… Hour.

Soldier:  I don’t know.

The Army trusts this Soldier to fire his rifle but he can’t figure out simple math or even understand a simple question that contains the answer to the question within the words used to pose the question…

Dude, Where’s My Car?

May 12, 2016 Army Stories No comments ,

One of my Soldiers, when getting out of the Army decided it would be best to move himself back home instead of having the Army take care of it.  Some people just like to get the job done themselves.  But he had a small car that wouldn’t be able to load up with much or tow a trailer, so he decided to look into buying a car.

He owed money on the car – a lot more than what it was worth for trade-in, so there would be negative equity involved if he traded it in.

So he got this great idea that he would just leave the car on Fort Benning.  Just leave it behind.  Despite everything everyone told him about how much of a bad idea it was…

He went out and bought the truck he wanted.  He also just stopped making payments on the car at about the same time.  In his mind, the respossession wouldn’t hit his credit until months after buying the truck and his payments wouldn’t be astronomical because he wouldn’t have a bunch of negative equity carried over from the car on top of the truck’s loan.

He left the Army and went home and his car was left abandoned in a parking lot near his former on-post house.  It’s not there anymore so I imagine it was repossessed.

I seriously hope he doesn’t want to buy a house for his family anytime soon.

Midgets on Unicylces

May 11, 2016 Army Stories No comments ,

Midgets on Unicylces

Some of the people I have served with have been… weird…  Weird to the point that they don’t live up to the definition of the word… they define the word.

Our company’s resident weirdo has a lot of odd stories he was involved with… this is probably the pinacle of his career.

It was a weekend, early evening.  A call was received from the staff duty NCO who had just come back from his first walk through of the barracks.  The NCO had called our company 1SG and the message flowed downhill…

Our weirdo had his window open and blinds open.  He was in his room, back to the window, sitting in front of his large flat screen TV.  On the screen was a video playing.. this video wasn’t pornographic in nature and was a video of midgets riding on unicycles.

Upon further inspection, the Soldier had his pants down and was masturbating.

The staff duty NCO said, “If you are going to do that you should probably close your blinds and your windows.”

The weirdo stood up, and turned around to provide the NCO a full frontal nude shot and said, “Oh, I was just trying to see if I could get off to this.”

Nothing ever came of it besides a counseling statement on monday morning instructing the soldier to close all windows, doors, and blinds if he wanted to masturbate.