Agile auditing in an IT project environment

Our Agile Internal Audit Journey article series began with Agile auditing: transforming internal audit to add greater value, where we discussed the role of internal audit, introduced the history of agile, addressed common misconceptions about agile auditing and set the baseline for the journey ahead. In the second article we explored applying the Agile manifesto and principles to internal audit. Then, we discussed Scrum, Kanban and agile project management methods applied to internal audit. In this article, we will build upon the information covered in the first three articles and explore benefits to performing internal audits of information technology (IT) environments that utilize agile principles and methods. Additionally, we’ll discuss audit considerations that should be assessed when performing system development life cycle (SDLC) reviews of agile IT environments.

The Agile Manifesto has transformed the way we do business and perform software development audits –  for the better. The Agile Manifesto, born out of frustration from IT project management teams, has been enhanced and expanded to apply to almost any business process or function. Agile principles are widely adopted in project teams as an effective tool for software development because they provide a viable alternative to traditional documentation-driven, heavyweight SDLCs. Agile principles have improved development team effectiveness by removing impediments to solution building.

In taking an agile approach to developing software, IT teams are more nimble. They can focus on responding to changing internal and external stakeholder needs while delivering software solutions to the market faster, in smaller increments. From an audit perspective, Agile has introduced advantages to the SCLC that has improved the transparency of tasks being performed and interactions between IT departments and the rest of the business. Controls-driven agile implementations in IT departments have introduced several benefits to internal auditors in performing assessments of project outcomes.

Individuals and interactions over process and tools

Development teams are more flexible and able to manage changing needs. Visual tools within most agile frameworks help to illuminate the progression of a project so that internal auditors can more clearly map a path between scope changes and a product owner’s business requirements. Scrum, in particular, focuses on congruent team activities, led by a Scrum Master, who builds controls directly into the development process that the team follows. Product backlog, sprints and reviews of the development team’s “definition of done” documentation help to map a path for auditors in a way traditional waterfall methodologies never could.

Working software over comprehensive documentation

Most regulations and standards require that key controls are documented and tested. Auditors recognize that documentation of key controls in software development cycles, while building system software, is important. Identification and documentation of controls improves the stability and security of the application, as well as enabling IT support teams to maintain and support these applications after they have been implemented in the business environment. To this end, agile-based projects have produced more relevant and usable documentation for IT project management assessments than traditional waterfall methodologies.

Scrum tools and techniques provide visualized documentation of product backlog and user stories that are incremental in nature, while overlapping sprints ensure that each increment is system-compatible and consistent with user stories. Auditors are therefore able to hone in on critical parts of feature-rich software builds and examine controls at a granular level to provide greater assurance of their effectiveness. Additionally, internal auditors no longer have to wait until an entire project is finished before performing a pre- or post-implementation review. Internal auditors may provide added value to business management by conducting assessments of early-stage product increments and giving timely feedback to project teams, which can then be applied to subsequent sprints. 

Responding to change over following a plan

Agile IT projects are driven by strong project management practices that were being used by Lean Six Sigma project managers even before the manifesto was adapted. Agile simply adopts a different approach to project management. Traditional waterfall objectives (design à build à test à implement à monitor) remain the same in agile. However, they happen iteratively in smaller increments: sprints.

In cases where there are multiple sprints happening for a specific project, key project management control points can be more effectively tested. The internal audit can select a representative sample of sprints and review these steps in detail. Or, auditors can understand the project as a whole by examining the first and last sprints. Ideal agile implementations provide many more control touchpoints, which allows auditors to gain comfort on the design and operation of system development controls. In the following table, typical agile control points and risks are discussed along with an approach for how they may be tested.

Customer collaboration over contract negotiation

Scrum is the most widely adopted agile methodology, and successful implementations are well controlled. However, as with waterfall, strong project management practices are sustained by a well-designed IT control environment. Project management controls alone are insufficient to mitigate the risk of unstable, insecure, poorly developed applications being introduced to the business environment. Influencers include strong IT general controls (ITGC), top-down governance processes, appropriate management of vendors and a secure infrastructure for the management of changes. These areas are critically important in ensuring that scrum results are accurate, stable and compliant with regulatory statutes.

Specifically, project teams should consider internal IT security teams as primary stakeholders. IT risk management and security functions should be engaged throughout the development process and be included in sprint meetings to ensure that secure systems and tools are being developed with compliant end-state configuration, roles and functions. As an added risk, Scrum teams also typically include one or several vendors who may have direct access to company information and a significant role on major projects. When assessing agile IT environments, the company’s vendor management and network security processes should also be assessed to ensure that proper controls exist to protect the firm and its resources.

DevOps as part of an agile audit

Development and operations (DevOps) accentuates agile IT processes through the provision of tools that automate the development of source code and autonomously manage its deployment to the production environment. Consistent with agile, DevOps seeks to improve the relationship with development and deployment teams that are typically physically and logically separated. Most DevOps projects are done in small product increments and typically employ either Scrum or extreme programming (XP) programming frameworks (both based on the agile methodology). The presence of DevOps in an IT environment should increase internal audit’s reliance on a robust change management process. Changes being deployed to production using DevOps tools (e.g., Jenkins) with source code pulled from shared repositories (e.g., GitHub) should be evaluated for security, compliance and congruence with business processes before implementation. See Figure 2 in the testing section below for a discussion on types of agile testing that should be deployed at various stages of DevOps implementations.

DevOps software testing is largely performed by automated tools, so internal audit should therefore ensure that user acceptance testing (UAT) is consistently performed as well. Typical DevOps groups include multiple vendors who may be providing access to shared source code repositories being used in the development of tools for the business. In each case, internal audit should consider who contractually owns the source code as well as any vendor access to sensitive business operations and information.

Agile principles: risk mitigation

Industry IT groups report that agile projects are 50% more successful than traditional waterfall projects and produce tools that are more consistent with changing client needs. The effectiveness of any project or process designed with agile in mind requires various external dependencies to ensure consistent project success. Agile projects are inherently variable and better suited for solutions where an end user  expects frequent upgrades and improvements, but in some cases, frequent incremental releases introduce additional risks to the stability of the control environment, enterprise brands and reputation. In these instances, an auditor should place further reliance on ITGCs with an emphasis on change management policies and practice. For example, internal auditors may choose to increase the number of samples tested for a change process with agile dependencies. In Table 1 above, we discussed various control points and documents within the Scrum framework and how to test them. Below, we present a list of enterprise vulnerabilities within agile IT environments and the specific agile documents or tools to use in assessing these risks.

Software development testing is more dynamic in agile auditing

One key area of risk in any SDLC audit is the process used for testing. Agile processes employ traditional unit, system, integration and UAT in new ways that enable a mixture of manual testing, automated code reviews and dedicated tools that scan and report application vulnerabilities. Software security and regression testing are performed using tools that scan and probe applications to reveal their vulnerabilities, and end-to-end system testing is performed by physically separated teams using a mixture of manual and automated scripts. There is more testing evidence available to auditors in different forms than ever before, and documentation of testing completion and signoff is more structured, and sometimes even automated.

The diagram below from Agile Alliance shows a typical agile testing quadrant for various test phases and its method of performance. Most agile IT environments will use a combination of methods to perform testing, so internal auditors should evaluate the adequacy of that testing method with relation to the inherent risks being tested. For example, if a security test was performed without tools, in a manual way, auditors may want to dig deeper into the effectiveness of the testing process.

Perhaps the most important test for an auditor to review is UAT. In agile, UAT may be performed in a variety of ways but should always include a specific signoff/approval. Flexibility and a sense of pragmatism should be employed when reviewing sprint and feature documentation to determine if the right controls are being tested, and at the appropriate phase in the development cycle.

Steps to completing an agile IT audit

Now that we have discussed ways to test key parts of the agile process and the inherent risks and vulnerabilities associated with some software development frameworks based on agile, here are a few steps to performing an agile assessment in IT environments.

Step 1: Understand the control environment

• Determine review objectives – Agile environments can be assessed in whole or in parts. Auditors may evaluate the design of the project management controls in an agile environment by evaluating multiple projects, or an auditor may choose to hone in on high-risk applications and tools that may be financially significant or have regulatory requirements.
• Map systems and data flows – Ensure that you understand the interaction between systems within the IT environment. In some environments, multiple methodologies may be used for different tools. For example, the company may have a mobile application that uses Scrum to deploy frequent updates, but using a sequential project development process for its financial applications. Auditors should ensure that where multiple methodologies are being used, all relevant controls are also tested. 

Step 2: Identify your risks based on the objectives of your audit

• You may be reviewing a specific IT project, system implementation, the SDLC or a combination of each. Reference Table 2 to identify specific business risks that are related to your audit and create an audit plan that addresses each risk using the guidance provided in Table 1.
• Ensure that the company has a robust vendor management practice that includes a right to audit, network security standards, non-disclosure agreements and service-level agreements. Review available System and Organization Controls (SOC 2) (SSAE 16) reports from vendors participating in development activities, especially if those vendors are service providers. In addition, the company should have network monitoring and prevention controls to anticipate and respond to unauthorized activities. How well a company documents and manages these processes will provide valuable insight on the state of the entire control environment.

Step 3: Consider implementing agile in your internal audit approach

• Part 3 of our article series applied agile to internal audit testing and showed how it can be used to manage large audit project teams with multiple controls to test. This method may also be used to perform agile testing of IT environments because it is effective in visualizing the scope and progress of the review, and it is scalable enough to accommodate testing for large and small organizations.

The typical agile implementation and transformation team is diverse, physically separated and always connected. These teams rely on a system of checks and balances within the agile process in order to succeed. The most successful implementations are those which produce highly stable, user-designed, compliant tools. Baker Tilly, has provided guidance and direction to many clients on implementing agile frameworks to manage SDLCs and redesign business processes.

For more information on this topic, or to learn how Baker Tilly specialists can help, contact our team.