by Martin Fisher

As we continue to discuss the Basic Truths of Incident Response Leadership, we’ve briefly gone over the three Basic Truths as well as done a deeper analysis of  “Succeeding By Planning to Fail”. This brings us to:

Basic Truth #2: Have A Workable Plan, or Else

As an Incident Response Leader, one of the most valuable parts of your role is to create, test, exercise, and (when called upon) execute Incident Response Plans (IRPs).  IRPs run the gamut from a Post-It note on the wall listing contact phone numbers, to plans that take up several 3-ring binders on a shelf somewhere.  Plans can be long or short, detailed or vague, paper or electronic, automated or manual…you get the picture.  What makes a good plan different from a not-so-good plan can be summed up in a few ways.

First, can you execute the plan using only the resources that you legitimately would have access to during the incident?  We’ve all seen plans that call for using network analyzers that aren’t accessible to the organization or that call for numbers of personnel that just don’t exist.  You may have written plans that assume that the responding team has skills and experience that your current team just doesn’t have (I have).  The key is to map out the current skills and capabilities of your team and employ them as best you can to meet the anticipated incident.

As you identify resources available to you, it pays to be creative.  Can other teams identify folks who could temporarily be available during an incident (think of it as an in-house “volunteer fire department”)?  Do you have relationships with designated outside incident response consultants? Do you have relationships with local, state, or federal law enforcement?  In today’s business environment, Incident Response Leaders need to be creative in identifying resources that can assist during a response cycle.

Second, you have to test the plan.  This sounds so intuitive, but many plans never get past the written-down stage before they are needed in an incident, because no leader stepped in to ensure that the plan would work as designed.  One of the most effective testing plans for an IRP is also the least expensive – the simple “Talk Through”, where all of the designated players sit at a conference table (pizza is optional, but highly recommended) and talk through the plan, noting any foreseen problems or issues.  The team needs to be encouraged to not only point out potential problems, but brainstorm solutions they can implement as-is since (as we talked about in Basic Truth #1) you can only plan on the resources you have, not the resources you want to have.

Plan testing needs to be redone each and every time the plan is modified, or at some regular interval (at least annually).  Testing can be announced or (my personal favorite) unannounced.  The time spent testing can help the Incident Response Leader assess not only the plan, but the team assigned to execute it.  The feedback loop should encompass applications, hardware, processes and procedures, as well as people.  Everything is fair game.

Lastly, you need to continually exercise your plan.  This, while not as intuitive as testing, is something that many organizations fail to do, claiming “it’s too hard” or “it’s too disruptive” or “it’s already been tested, why should I do an exercise?”  Having performed incident response on plans that have been exercised and plans that have not, I can tell you with complete assurance that plans that have been exercised are executed more smoothly, with fewer problems and a better resolution.

Exercises can range from a talk-through (similar to testing but without the constant feedback loop) to a full-on exercise using live equipment.  Talk-through exercises can help in quickly familiarizing a team with a new (or newly updated) plan.  Talk-through work will also quickly point out assumptions that, while seemingly accurate in testing, don’t fit the way the incident response team works.  All other things being equal, I believe that talk-through exercises offer the highest return for time spent in any aspect of prepping for a incident.

Full-on exercises, as powerful and complete as they are, can be very hard to accomplish.  Most organizations cannot fully replicate their production systems (even using virtual machines).  These exercises, when they can be done at all, are usually done in development or test environments and generate most of their value by allowing teams to actually assess and interpret adversary actions and data.  These exercises are an Incident Response Leader’s best chance to simulate the stress and activity of a real incident.

Taking all of this into account, it’s clear that the Incident Response Leader must be able to create, test, and exercise an IRP to be able to effectively respond during the inevitable incident.  By creating plans designed around available resources, qualifying the plans with testing, and regularly exercising the plan, you can ensure that you and your organization will be ready when the inevitable incident occurs.

But it’s not over yet. Once you’ve gotten this far you still have one vital task to accomplish.  We’ll cover that in the last article on the Basic Truths of Incident Response Leadership.

By Ron Simmons

Do you have a documented and tested incident handling program? To my surprise, some a majority of companies lack this basic necessity. Putting something in place may take some time, but here are some tips and suggestions to help get started.

Define: incident

For those familiar with ITIL (Information Technology Infrastructure Library), it is important to note a security incident is not the same as an ITIL incident. ITIL describes and incident as and event that is not a part of the standard operation in these terms an incident is the start of a service delivery or problem management process.

While in security events are the starting point for us, they are issues in the enterprise that could lead up and incident. The best definition of incident that I have read out there is “An incident is any situation that exceeds normal risk management processes.”

Instead of focusing on a specific type of incident, an “Incident handling” process must be designed to be broad enough to support any type of threat or exposure to organizations systems, data, and/or physical infrastructure, etc…. Breaches happen in many forms now days, improper disposal of documentation, lose of equipment, system compromises, all the way to the janitor attempting to sell DOE restricted data. However when you think about it, an incident may not end up in a breach, but a breach will always be an incident. Breaches encompass many forms of data, personal information, financial account information, health care information, trade or governmental secrets, and the list goes on.

Simple is always effective, and it is no different when it comes to incident handling. Good incident handling does not require a novel. To get started, keep terms general, include process flow charts and check lists. When responding to an incident, these will keep everyone on track and ensure all actions are documented.

The process can be broken down in 5 simple steps:

 1.   Prepare (for incidents)

 2.   Identify

 3.   Incident handling

 4.   Recover (from incident)

 5.   Lessons learned

Preparation – Proactive

Preparation is the most critical element. Document the guidelines to handle incidents. Preparation ensures you will be better equipped to handle most situations as they come up.

First start by identifying what your organization has that it considers critical assets. This could be technology, customer data, physical assets, even services that are offered. Then you need to identify what types of threats does your company face. Keep it simple here, no need to go into specifics, IS threats, physical threats, insider threats, etc…. Then document your policy, guidelines, and checklists on how to identify, resolve, and recover from them.

Remember to keep it general, there is no way you can identify all possible threats down to exacts.

Identification

This is the most difficult step of this process. The ability to gather “events” is crucial – without it, the incident could be missed. An organization has to know the sources of the events and what their meanings are in order to validate and event and move into the incident handling process. The best piece of advice anyone could get here is “log… and log everything.” Without logs you can and most likely will miss the event that leads to your incident. The last thing anyone wants is for an outside company to discover the incident before you do as a company.

To help you could come up with cheat sheet to assist your employees in identification of incidents.

Incident Handling

Here is where we stop the bleeding. Once the incident has been identified assemble your strike team, move in, and eradicate the incident. Here is where preparation comes in:  if the documentation is in order, including the procedures on how to handle different types of incidents, then dealing with them in a rapid and efficient way will be a snap.

Recovery

“Simply” put things back to normal. Make sure that back to normal does not include the risk that was abused in the first place to create this incident. In a cyber incident this normally means rebuild from scratch. Be careful if you decide to restored from backup, if can not or have not identified exactly when the breach or incident happened, you could restore back to the bad state and have to start all over. It is best to restore your configurations from your configuration management system.

Lessons Learned

Take what you have learned from this and return to the preparation phase and make modifications as necessary. These modifications could also mean changes to the production systems, the physical environment and even the training you give your employees. Sometimes these changes could seem silly, but you still need to heed the warnings and make the change.

http://www.knowledgetransfer.net/dictionary/ITIL/en/Incident.htm

http://securosis.com/2008/08/20/the-best-incident-response-training-you-can-buy-for-free/

http://darkreading.com/security/attacks/showArticle.jhtml?articleID=212902962

Happy Monday! The forums have really seen an uptick in membership and activity in the last few weeks. This is a supportive environment where professionals come together to ask for help, share ideas and get validated. Here is some recent activity (and darn good discussions):

• Incident Response Case Study: Shutdown the Network?
• Protocol Security: Where does it belong?
• Web Server vs Reverse Proxy
• PCI DSS clarifies 6.6 Requirements
• IPS Matrix

I spent a great day in Rochester, NY yesterday. Here is some of the activity in the forums  - check it out to add your opinion or learn (lots here to learn from):

• Porn Scanner
• Reporting Incident Response Statistics
• Vulnerability Management Process/Workflow
• The cost of PCI compliance — or non-compliance — for small organizations
• DFRWS and OMFW