Penetration Testing Methodologies Critique

In my experience, the cybersecurity field is fairly immature for the age of the field of study. I’m frequently surprised by the lack of generally accepted practices and standards within the field. One particularly important and underdeveloped segment of the industry is penetration testing methodologies. Below I intend to give my review of four penetration methodologies that exist today and delve into on a high level what works in these methodologies and what does not. and try to give some guidance on which methodologies are most useful for an organization that is looking to implement a stronger security practice. While rating these methodologies I intend to rank them in a manner which compares them to one another which does not account for deficiencies that I believe they all share but I believe this shows a relative quality check for individuals looking to find the best methodology that is currently available.  

To rate the four methodologies I’ve reviewed below I tried to use a standard group of rating metrics to be as fair as possible between my ratings however some of the metrics are partially subjective and some are biased towards quantity of datas over quality of data. I utilized a 5 point ranking system 1 being the lowest ranking and 5 being the highest ranking for each category. The final score give to each methodology is the average ranking for all the ranked criteria. The 13 criteria I used for my rating system is as follows: Availability of training associated the with methodology, Depth and completeness of the source documentation, Age of the documentation, Broadness applicability to multiple applications and systems, Does the methodology provide recommendations to specific areas of testing, Does the methodology provides specific recommended tools, Quality and Standardization of reporting, Accreditation of organizations performing the tests, Ease of implementation with a limited skilled team, Availability third party supplemental training or books, Industry acceptance based on 50 consultant survey, Google search results for the full methodology name.Access to my data recorded for this review as well as the raw rating score data can be found in the following spreadsheet. Research Data

The National Institute of Standards and Technology (NIST) is a sub department of the US Department of commerce. NIST creates a number of  Standards within the field of Computer science and they have a fair amount of clout with in the technology industry. While not dedicated solely to the Cyber Security industry NIST created the Technical Guide to Information Security Testing and Assessment Special Publication 800-115 (NIST 800-115). The purpose of this publication was to create a standard security testing methodology for the federal government and its agencies to ensure their computer systems are adequately protected from cyber threats. While not designed for the private sector NIST 800-115 is broadly applicable to many organizations. Additionally carrying the name of the of NIST provides this methodology a fair amount of clout within the industry even if you haven’t heard of this special publication you have surely heard of NIST and would likely put some confidence in its methodology with no other information.

NIST 800-115 is a fairly flexible the methodology itself is applicable to most types systems and applications and will likely continue to work for the foreseeable future. The Methodology makes some specific recommendations in the area of network attacks and password attacks these are likely some of the most common security concerns and organization may encounter so it's a reasonable area to dedicate some additional focus. NIST 800-115 also provides a list of some of the most important tools to consider using when engaging in a penetration test. Finally NIST 800-115 has one of the most comprehensive reporting templates of all of methodologies I’ve reviewed here. The reporting template lists out every aspect of the penetration test it covers who’s involved, what was tested, when it was tested, and what were the outcomes of the test as well as mitigation and remediation recommendations.

Since NIST 800-115 was not created for the private sector and the organization that created it doesn’t have any goals of advancing the quality within the private sector there are a number of training and accreditation processes that are lacking for NIST for private sector companies looking to utilize the methodology. While I’m sure the US government provides NIST 800-115 training internally to Government penetration testers. This training is not provided to the general public there are no NIST Certification course for private sector penetration testers to demonstration the knowledge or compliance with the testing methodology. Additionally there is no process for an organization to gain accreditation or certify that their organization is in compliance with the NIST standard. Another impediment to the broad acceptance of  the NIST Special publication is, outside of the government provided documentation, there are few additional learning resources for the methodology an amazon search for additional books on the subject yields only four results which is better than some of the other methodologies reviewed but not a large breath of resources to choose from.

NIST 800-115 is also an aging methodology the special publication was released in 2008. While the age of this methodology may be less of a hindrance for the public sector since it is notoriously behind the times in terms of technology in many of its departments. This is a big impediment for the private sectors use of the methodology. While you can still use this methodology for modern systems one need look no further than their numerous references to the tool Backtrack (now known as KALI) to know you will need some modern security knowledge and will need to fill in a lot of the decades old gaps in their recommendations to be successful with this methodology. This need to reach out to more modern resources to supplement the methodology also means your team is going to spend more time researching more modern approaches which is going to make this methodology adoption speed likely a little slower into a private sector organization than some of the other methodologies. 

NIST 800-115 is very thorough and frequently used within the public sector. NIST has a long history of providing standards that are utilized and respected within the industry as a whole. However, the methodology suffers from being rather out of date for the landscape of the private sector. There is also a severe lack of training for the private sector on this methodology which makes it difficult of this methodology to gain a strong usage outside of the public sector where it is required. All this taken into account I’ve given the methodology a rating of 3 out of 5. 

The Payment Card Industry Security Standards Council (PCI) is an international community of the credit card and banking organizations. The community created a penetration testing guide as a part of the PCI Data Security Standard (DSS). Penetration tests conforming to this guide must be performed by any organization processing credit card payments on the systems that perform the credit card processing and any application or system connected to those credit card processing systems. As this is required by most major credit card companies to be allowed to process their credit cards this is likely the most commonly used penetration methodology in the private sector. Most people within the industry have likely heard of the PCI security requirements and lend a fair amount of credibility to any organization performing these types of tests. Of the 50 consulting firms surveyed 23 claimed to provide PCI compliant Penetration tests.  

PCI provides nine certifications associated with their security standards. As well as numerous supplemental documents and resources. PCI DSS is also supported by numerous third party organizations. An amazon search for training books on PCI DSS compliance will yield hundreds of additional resources. If looking for a strongly supported security framework one need look no further than the PCI DSS. Due to its large industry usage there is lots of additional material to help train individuals on it. Since this is a highly utilized methodology and it is also the most modern and recently updated methodology of all the reviewed methodologies. Finally PCI DSS has a fairly consistent and straight forward reporting standard. These reports are often highly detailed as they need to be repeatable in the event of an audit by PCI. While there are not accreditation programs for organizations providing these tests non compliance with this standard can result in removal of a company's ability to accept credit card payments which puts a high level of scrutiny over organizations performing these tests.

One of the big draw backs of this methodology is that it is highly focused on credit card payment systems and applications while the methodology itself could be applied to any industry or system. PCI training is likely to have a payment card system focus that may not be as useful to a team that has no interest or compliance requirements related to this. Also many of the more specific recommendations if the supplemental documentation is likely geared to Payment card systems and their associated systems. The primary documentation of the methodology itself is fairly brief which does leave something to be desired. These gaps are filled in by many of the supplementary documents but if your desire was to use this methodology outside of the PCI industry you would likely find the supplementary documents to be less applicable to your systems. 

One other big drawback of using the PCI testing guide is that it makes few technical recommendations for implementers. So someone utilizing the PCI testing methodology must come to the table with more technical penetration testing skills already. If you are looking for a methodology to use to get your security practice started this is a high barrier of entry if you already had someone skilled in penetration testing skills they likely already know a penetration testing methodology other than PCI and can get your security practice started quicker without using this methodology.

PCI testing guide is probably the most widely used methodology for penetration testing in the world. It has a large breath of documentation and training resources. However it is highly focuses on one specific part of a companies systems and applications. It’s highly unlikely that any company is using the PCI testing standard for any system it isn’t required to do so for compliance reasons. Additionally it’s a fairly high standard to adhere to many organizations would find its requirements hard to apply to all of their systems and applications. Due to all of these factors I’ve given the PCI Testing guide an overall score of 3 out of 5.  

The Open Source Security Testing Methodology manual (OSSTMM) is and open source Penetration testing methodology created by the Institute for Security and Open Methodologies (ISECOM). ISECOM is an international organization of researchers and industry experts dedicated to the advancement of security testing and implementation. ISECOM exists almost exclusively for the furtherance of their methodology this is in contrast to the other methodologies discussed previously who’s methodologies are created as a small subset of larger security initiative or practice. This high level of focus on just the methodology is evident when reading the methodology manual its highly developed and well thought out. The methodology has well developed reporting templates and highly structured and objective testing recommendations. However, Lacking a large body of other security documentation and recommendations within the industry also lends less credibility to the ISECOM and their methodology. Outside of the methodology space it would be easy to have never heard of ISECOM, NIST and PCI do not suffer from this problem as they are well known within the industry for their presence in a large number of non methodology related spaces.

The OSSTMM provides an good framework for penetration testing. The Manual itself is over 200 pages and is highly developed. The manual provides a great background of high level concepts to understand in order to perform thorough and well thought out penetration tests. The framework is fairly broad and could be applied to multiple application and environment types. The Framework also provides some more specific recommendations on specific venues of attack such as  Physical security, Human Security, Data Network and Wireless Network testing. Additionally, ISECOM offers six different trainings and certifications associated with their testing methodology. ISECOM also has a licensed auditor program which in theory should lend additional credibility to companies that are licensed and help distinguish those organizations from less developed or lower quality organizations. Finally the methodology has a standardized reporting template and scoring system. This helps make the results of a OSSTMM penetration test objective and consistent between different organizations.

Despite having a highly developed methodology and fairly strong training and certification backbone the OSSTMM has many shortcomings. When taking a deep dive on their website the cracks start to become apparent quickly. The OSSTMM seems almost abandoned by its creators. Their website is need of an update, pages still reference the previous year four months into the new year. Despite multiple attempts email addresses on the website do not respond to inquiries. Their last update to the OSSTMM is from 2010, a pretty significant amount of time in terms of a quickly evolving field like computer science. 

Another shortcoming of the OSSTMM is the lack of supplementary or third party guides and training. This highlights two key failings of OSSTMM. industry acceptance and third party support. If OSSTMM was a more utilized methodology within the industry one would expect to find professionals in the industry creating their own derivative works based on their use of the methodology adding to and improving the methodology with their perspectives and experience. However a amazon search for additional books or guides on OSSTMM yields only one book on the subject published by ISECOM. Additionally of 50 penetration testing consulting firms surveyed only two noted that they used the OSSTMM for their penetration tests.

Overall OSSTMM has a good foundation provided by ISECOM. OSSTMM had potential to elevate the quality of the security industry as a whole. Unfortunately, OSSTMM never seems to have caught on within the industry and every day that its not getting industry support is another day it gets more outdated and less applicable to the current security landscape. Given all of this my rating for the OSSTMM as a methodology is 3.5 out of 5.

Similarly to OSSTMM the Penetration Testing Execution Standard (PTES) was created by an international group of information security professionals. Their goal was to provide a standardized methodology to to help improve the quality of Penetration testing within the industry. Unlike OSSTMM, PTES is much more operational and technical focused than OSSTMM. It’s also highly geared towards security consultant organizations. Additionally much like OSSTMM, PTES suffers from having limited impact outside of the Methodology  space in the security field as I touched on earlier this limits the credibility of the methodology when compared to the methodologies of NIST and PCI which have a larger broad influence in the industry.

PTES is a very streamlined methodology the manual for it is only 80 pages of material and that includes technical recommendations like what tools to use and when. The documentation for PTES is starting to get a little dated it’s not as outdated as OSSTMM however it has been about 4 years since their last update to the methodology. Its a quick down and dirty step by step guide to performing a penetration test. While nothing in the methodology would be wholly useless to an internal security team performing a penetration test there is a higher focus on limiting liability, establishing communication, and payment terms which is much more important to a consultant rather than an internal team. In terms of high level concepts PTES doesn’t  bogged down in a lot of jargon or theory it gives you just enough to get a penetration test done effectively and quickly. It's a really good roadmap if you are starting a small independant penetration testing consultant company of 1-6 people. 

Unlike OSSTMM, PTES lacks a lot of the legitimacy backbone that OSSTMM could provide to its methodology. For example PTES lacks any training or certification process above and beyond the documentation on the site. PTES also does not have any accreditation process for organizations that are using their methodology, making it hard to differentiate those that claim to use the methodology and those that strictly adhere to the methodology. 

While PTES’s streamline approach allows small organizations to get up and running with a workable methodology quickly the brevity of the methodology also leaves something to be desired in the way of completeness and depth. The methodology leaves a lot of gaps that an implementer will need to fill in on their own. The gaps in the methodology coupled with the low barrier of entry to use the methodology would likely lead to more fragmentation in the industry, rather than more standardization as the designers envisioned, if PTES had a larger presence in the industry. However, much like OSSTMM, PTES has failed to make a big impact in the Security community only 1 out of 50 penetration testing firms say they use PTES. A quick google search of their methodology only returns 136,000 references to the methodology in webpages when compared to OSSTMM or NIST 800-115 who’s results number in the millions it’s easy to see PTES has limited recognition or use in the industry. 

PTES is an excellent starting point if you are looking to create a security practice to sell or to even if you are trying to start a internal security practice and don’t know where to start. However, as your practice evolves you may find that PTES doesn’t provide enough structure for a mature security practice. Additionally the PTES methodology likely does not carry with it a well know industry name that will lend credibility to your organization and the quality of your penetration tests. Therefore I rate the PTES methodology 2.5 out of 5. 

In Terms of picking a definitive best methodology on the market I can’t in good conscious recommend strict adherence to any of these methodologies. Each of them lacks an important component that the others are strong in. For the practical reader of this essay the best recommendation I can make is to pick and choose from all four of these methodologies. Unfortunately in terms of standardization this means my recommendation is avoid standardization. Which leads to more fragmentation and less consistency within the industry which is roughly where we are as an industry today. If we return to my survey of 50 consultant organizations the number one representation by 46 of the 50 consultant organizations was that they use their own proprietary methodology based on the above reviewed  methodologies as well as many more. Clearly the industry needs a standardized methodology to provide consistency and quality. However, no vendor can provide both to their customers using existing methodologies without significant supplemental training and effort meaning for the vast majority of companies in the industry there is currently no viable option that provides both.