Risk Assessment of Information System Audit Tools Based on SPBE Risk Management Guidelines

The SPBE Risk Management Guidelines adopt SNI ISO 31000. This standard is used because it is repetitive in risk management and can assist organizations in strategy formulation, goal achievement, and decision-making. Table 1 presents a comparison between SNI ISO 31000 and Permenpanrb No.5 of 2020 as follows:

Table 1. Comparison of ISO 31000 with Permenpanrb No. 5 of 2020

Risk is the effect of uncertainty in achieving a goal that can have positive, negative, or both impacts [22]. Risk is a combination of the likelihood of an event occurring and the impact of that event [23]. Risk management involves identifying, assessing, and controlling threats to an organization's assets and operations [24]. In the implementation of digital technology, risk management needs to be conducted. The rapid changes in technology make organizations quick to respond and adaptive.

Many risks arise from the transition from manual service systems to electronic ones. Therefore, the government mandates SPBE risk management for all IPPDs. SPBE Risk is the opportunity for events to occur that can affect the success rating of SPBE implementation [21]. A structured procedure is carried out to determine the best solution for SPBE risks, starting from the process, measurement, structure, and culture of SPBE risks. Identification, analysis, control, monitoring, and evaluation of SPBE risks are systematic processes of SPBE Risk Management, as outlined in the SPBE Risk Management Guidelines. The SPBE Risk Management process can be seen in Figure 2.

2.png

Figure 2. Risk management process

2.1 Establishment of SPBE risk

In establishing the SPBE risk context, the basic parameters and scope of SPBE risk application to be managed will be identified. In this process, the determination of SPBE risk categories, identification of SPBE risk impact areas, establishment of SPBE risk criteria, creation of an SPBE risk analysis matrix, classification of SPBE risk levels, and determination of SPBE risk appetite will be carried out.

2.2 SPBE risk assessment

The following process is risk assessment, which involves identifying, analyzing, and evaluating SPBE risks [7]. The goal is to understand the causes, likelihood, and impact of SPBE Risks in IPPD. The following are the stages of SPBE risk assessment, namely SPBE risk identification, SPBE risk analysis, and SPBE risk evaluation.

Risk identification is divided into two types: positive and negative risks. Positive risks can enhance the success of achieving the goals of SPBE. Meanwhile, negative risks can hinder the achievement of SPBE objectives. Then, a risk analysis is performed by selecting the control system, likelihood level, impact level, risk magnitude, and risk level. The final stage of risk evaluation is to decide whether the risk will be addressed according to the handling priority.

2.3 Handling SPBE risk

Risk prioritization and risk handling plans are determined in the risk management process. The risk management plan is determined by creating risk handling options, risk handling action plans, outputs, and responsible parties.

In this chapter, the application of the SPBE Risk Management Guidelines in the management of SPBE Audit Tools will be discussed according to Figure 3. Here are the results of the implementation:

4.1 Establishment of SPBE risk context

Based on the explanation in the previous section, the first stage in SPBE risk management is establishing the risk context of SPBE. At this stage, it consists of target identification, risk category determination, risk impact areas, risk criteria establishment, risk analysis matrix, risk levels, and risk appetite from the management of SPBE Audit Tools.

In this research, eight categories of SPBE risks will be used, namely, system development or enhancement projects, data and information, SPBE infrastructure, SPBE applications, SPBE security, SPBE services, SPBE human resources, and natural disasters [21]. As for the impact of SPBE risks, it consists of financial, reputational, performance, organizational service, operational and ICT asset, as well as legal and regulatory consequences.

Then risk anlysis matrix (Table 2), it is based on several regulations such as Permenkominfo No.16 of 2022 [26] and BRIN Regulations No.1 of 2024 [27].

Table 2. Risk matrix

Next, determine the risk level of the Audit Tools [21], as follows: Table 3.

Table 3. Risk level

The final stage is determining the risk appetite, which aims to show the order of priority in handling risks by considering their positive or negative aspects. For positive risk, the minimum handling value is ≤ 10. Then, for negative risk, the minimum handling value is ≥11, except for human resources and natural disaster categories with a minimum of ≥16.

4.2 SPBE risk assessment

In this section, the Audit Tools Application management risk assessment will be conducted according to the SPBE risk management stages.

4.2.1 Risk identification

The first step is to conduct a risk identification. Risk identification is obtained from the results of observations and interviews. Additionally, it considers previous research on information system risk management in government.

Table 4. Risk identification

Based on the risk identification in Table 4, it can be seen that there are 27 identified risks divided into eight risk categories. The risks mentioned above are a combination of positive and negative risks. Some are risks that have occurred in the Audit Tools application.

Then, some risks were also identified in previous research. Such as the risk of data access abuse, server disruptions, power outages, network connection issues, limited IT resources, cyberattacks, and natural disasters [15-17, 19]. Governance and management are also important factors in the successful implementation of SPBE. Therefore, risks related to the governance and management category have also been added.

4.2.2 Risk analysis

The first risk analysis begins by dividing risks into positive and negative risks. The probability value and impact value are obtained from the questionnaire results. The risk value is the result of the matrix of probability values and impact values from Table 2. For the risk level based on the explanation from Table 3.

The analysis in Table 5 reveals that out of the 27 identified risks, six are positive and 21 are negative. As explained earlier, this means that 78% of the identified risks hinder the success of Audit Tools' strategic objectives. The results of the risk analysis, as presented in Table 5, are more clearly illustrated in Figure 4.

From Figure 4 below, it can be seen that there are five high-level risks in positive risks, namely planned system development, data access, information security by policy, audit services by regulation, and rapid response from audit services. This means that the Audit Tool Management Team has followed the existing rules regarding the Development of SPBE Applications. The Audit Tool Management Team follows the planning for implementing the SPBE Audit as described in the SPBE regulations.

Table 5. Risk analysis

4.png

Figure 4. Risk level of audit tools

Then, among the existing positive risks, there is one very high-level risk, which is the increase in users of the Audit Tools Application. This is due to the obligation of all IPPDs to conduct ICT SPBE Audits. Moreover, an API exists between the SPBE evaluation application, Tauval, and the Audit Tools application. Data on the implementation of application and infrastructure audits will be directly retrieved as supporting evidence for the audit execution. The implementation of the SPBE Application and Infrastructure Audit adds value to the SPBE evaluation.

Out of the 21 identified negative risks, there are five high-level risks: system development being hindered, server downtime, slow access, outdated technology, and the Audit Tools Application management team being overloaded. Those five risks are indeed the ones that have been occurring in the management of the Audit Tools Application. As explained above, server downtime and slow access often become obstacles in implementing SPBE Application and Infrastructure Audits. Moreover, if the audit completion deadline is approaching. This will impact the completion of Audit implementation.

Then, regarding negative risks, there are 14 medium-level risks, namely no data access, data and information are not transparent, and power outage. Other risk are network monitoring is inadequate, user satisfaction level is not good, access to the application besides the auditee and auditor, cyber attacks, and physical attack on the data center. Besides that, there is a risk that audit services do not comply with regulations, the slow response of the Audit Service Team, the HR capabilities are inadequate, the infrastructure managers are not working, fire, and earthquake.

The above risks are common in the management of information systems in government. These risks will have a significant impact, namely, on the reputation of Audit Tools. The trust in IPPD decreases due to the many unresolved risks. Then the performance of the Audit Tools management will also decrease. The operation of the application will be disrupted, resulting in a delay to the audit process. The implementation of the SPBE Application and Infrastructure Audit is time-bound.

In the implementation of an audit, data and information are exchanged between the auditee and the auditor. For data access and transparency, data and information must be accessible to both the auditee and the auditor. Audit account access must be ensured not to be leaked, as audit results are highly confidential for IPPD. IPPD's trust is at stake from this data access.

The occurrence of cyber attacks, physical attacks on data centres, fires, earthquakes, power outages, and inadequate network monitoring will disrupt the ongoing Audit. The web application slows down and even becomes inaccessible, which indeed halts the audit process. This affects the application's satisfaction level, especially if it disrupts the supporting data uploaded to the application.

Application and Infrastructure Audit Services must be conducted by applicable regulations. So that the audit implementation runs according to its business process. The ability and speed of the management team's response will significantly impact the timing of the Audit. Therefore, the Service Team on duty must promptly respond to emails from IPPD.

There are two risks at a low level for negative risks, namely, audit data leakage and flooding. Audit data leakage will impact Audit Tools' reputation, leading to a decrease in users and a decline in IPPD's trust. For floods, the management of Audit Tools is already in a flood-free condition. So it has a low level. Here is a summary of the risk levels from the 27 identified risks. In Table 6. below, it is a combination of negative and positive risks.

4.2.3 Risk evaluation

After that, based on risk appetite, it will be determined whether the risk will be addressed or not. From Figure 5, it can be seen that none of the positive risks were addressed. Then, for the negative risks, 6 risks were not addressed and 15 risks will be addressed. The six risks that will not be addressed are audit data leaks, inadequate IT personnel capabilities, non-functional IT personnel, and risks associated with fire, flood, and earthquake.

Table 6. Audit tools risk analysis results

5.png

Figure 5. Handling risk

4.3 Handling on risk

For all positive risks, risk acceptance will be carried out. Because all positive risks already have a positive impact on the management of the Audit Tool Application, such as improved reputation, performance, service, and operational efficiency of the Audit Tool. For negative risks, the priority of each risk is determined. The top five risk priorities are server downtime, slow access, hindered system development, outdated technology, and cyber attacks. The risks that will be escalated are system development and technology updates for the Audit Tool Application. Risk escalation is the transfer of risk responsibility to higher authorities. Because the core Audit Tool Management Team currently lacks the necessary human resources for application development, it needs to be transferred to the relevant work unit. Other risks will be mitigated through risk management.

4.4 Maturity level

In addition to implementing risk management according to the SPBE Risk Management Guidelines, this study also includes the measurement of information system maturity [13]. Data from 8 respondents who are the Management Team and users of the Audit Tools, the existing risk categories were analyzed on a scale of 1-5 by all respondents.

The purpose of calculating this maturity value is to assess the organization's maturity in implementing the information system according to the eight risk categories. From Figure 6 above, it can be seen that the target maturity for the risk category is valued at 5. The application category has the highest maturity level at 3.28, followed by the system development category with a score of 3.25. The IT human resources category has a maturity score of 2.91, while the infrastructure category has a score of 2.66. In the service category, the score is 2.46, in the data and information category, the score is 2.41, and in the information security category, the score is 2.25.

6.png

Figure 6. Maturity level of audit tools

The category with the lowest maturity value is natural disasters, which is 1.5. The average maturity score is 2.59. There is a GAP between the target and what has been implemented, which is 2.41. There is a GAP between the target and what has currently been implemented. Therefore, recommendations are needed for the management of Audit Tools. From the results of the risk analysis and risk evaluation, the best handling will be provided for each risk. So that in the future, the maturity of the Audit Tools Application will continue to improve.

4.5 Recommendations

Recommendations will be formulated from the results of risk identification, risk assessment, and risk handling above, as well as the results of interviews with the Audit Tools Development team. Here are the recommendations for the management and development of the Audit Tools Application:

4.5.1 System development

Coordination between the Audit Tools Management Team and the National SPBE Team is necessary in system development. Because the implementation of the SPBE Application and Infrastructure Audit is based on the applicable policies. In addition, good planning is needed to develop audit tools according to the needs of the users, namely the auditee and auditor of each IPPD.

4.5.2 Data and information

In the category of data and information, there are risks related to data and information access, lack of transparency in access, and data and information leaks. To prevent those risks from occurring, the Management Team must create an Audit Account Control SOP. Each IPPD only provides the username and password of the account to the responsible auditee and auditor. For user account access, SSO can be used because it enhances security through the use of additional authentication methods, such as two-factor authentication. Additionally, SSO can simplify account management [3].

4.5.3 Infrastructure

The infrastructure category has risks such as server downtime, slow access, power outages, and an inadequate network. To avoid those risks, it is mandatory to conduct an infrastructure audit annually to assess the capability value of the Audit Tools management infrastructure. Hiring cloud services from third parties has certainly been done, but regular monitoring of cloud usage is necessary. Then, if an incident occurs or there is infrastructure maintenance, the Audit service team must immediately notify the audit website. In addition, coordination between the Audit service team and the infrastructure team must be carried out. This is done to prevent disruption of the services and operations of the Audit Tools.

4.5.4 Application

In the Application category, there are risks related to user satisfaction levels, the number of users, application access, and application technology. To increase the number of users, conducting periodic socialization of the SPBE Application and Infrastructure Audit Implementation to all IPPDs is necessary. In addition, the application socialisation also needs to include information about the risk mitigation plan, so that they are aware of the application's cybersecurity policies [28].

 Implementing the SPBE ICT Audit complies with SPBE policy, which needs to be made a culture by all IPPDs. Training on application usage will also increase user satisfaction levels. The features in the application need to be improved and made user-friendly. For example, the additional question feature needs to be reviewed or removed. Because so far, it is still error-prone, and no IPPD auditors have added questions outside the provided question indicators. Then the feature to add audit activities by auditors should be removed. Because its creation is carried out by the Audit Services Team. There are still many other features that need to be maintained and reviewed. The application features need to be periodically improved for user comfort.

4.5.5 Information security

In the information security category, there are risks of cyber and physical attacks on data centers. To prevent such risks, there needs to be an Information Security policy. In addition, it is necessary to conduct a Security Audit every year. The Information Security Team must continue to work according to their duties to prevent cyber and physical attacks. To address the risks to data centre security, organisations can perform real-time backups. Additionally, they should update to the latest monitoring tools [11].

4.5.6 Service

In the service category, there are risks of non-compliance with regulations and slow service response. Therefore, forming a solid service team that can quickly respond to service requests is necessary. The Audit service email needs to be monitored to respond promptly to service requests. In addition, periodic evaluations of the SPBE audit services are necessary. The service website needs to include a help desk and a user manual for the service so that IPPD does not get confused when they want to request an audit account. The preparation of the service SLA must also be carried out.

4.5.7 IT human resources

In the IT HR category, it is necessary to periodically enhance the competencies of the Audit Tools management team. Conducting training sessions as needed, such as Audit services, incident and disaster handling, system/application development, etc. [29]. In addition, personnel placement according to needs must also be carried out. To improve service and performance, increasing awareness among the team must also be done. After that, it is necessary to conduct periodic performance evaluations of the Audit Tools management personnel to ensure operations run smoothly without any incidents occurring.

4.5.8 Natural disaster

In the disaster category, there are fire, flood, and earthquake risks. The presence of cloud in the data center will help avoid those risks. In addition, it is necessary to create a Disaster Recovery Plan so that each personnel member managing the Audit Tools understands how to recover the system in case of a disaster. By conducting monitoring and evaluation of cloud and physical infrastructure, the services and operations of Audit Tools will run smoothly. There is a need to develop earthquake-resistant physical infrastructure and update disaster early warning systems [30].

The datasets analyzed are not publicly available due to their confidential nature and institutional protocols. This paper considers the data confidentiality restrictions set by the institution.