Johanes Fernandes Andry*
| Hadiyanto | Vincensius Gunawan
© 2023 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
In the increasingly competitive landscape of the manufacturing industry, leveraging advancements in information technology has become pivotal for gaining a competitive edge. Specifically, the furniture manufacturing industry has adopted decision support systems (DSS) to meet diverse information needs. However, user concerns regarding these systems, including information reporting speed, order reception, and the accuracy of raw material usage calculations, persist. The development of any system necessitates the application of quality assurance standards that align with user requirements. In light of these concerns, the measurement of software quality emerges as an essential strategy to assuage user doubts about the DSS. Employing the ISO 9126 standard as an assessment parameter provides a reliable framework for evaluating the quality of the DSS. Although the ISO 9126 model possesses several characteristics, this study focuses exclusively on two: functionality and reliability. The anticipated outcome of this research is a refined understanding of user satisfaction pertaining to the implementation of the DSS. Consequently, this study's findings may contribute to the DSS's credibility, specifically concerning the speed of reporting, order reception, and the precision of raw material usage calculation. This, in turn, can influence the decision-making process in production, thereby enhancing the overall effectiveness of the manufacturing industry.
ISO 9126, decision support system, user acceptance testing
The contemporary era is characterised by the rapid proliferation of information technology, with its influence permeating virtually all domains [1, 2]. These technological advancements, acting as an indispensable tool for business processes, provide an integrated real-time system to compile and furnish a diverse range of accurate data [3]. The strategic adoption of information technology is considered a cornerstone for gaining a competitive edge in an increasingly competitive landscape [4].
However, the development of information systems in industries transcends the mere automation of routine business processes. The objective herein lies in the orchestration of a novel information flow, systematically integrated into a cohesive system [5]. This premise holds particularly true for the manufacturing industry sector, where advancements in information technology have catalysed the emergence of innovative business models aimed at expediting processes and optimising time efficiency. In this context, the focus is predominantly on industries engaged in goods production.
Identified as a pivotal sector in modern economic and business development, manufacturing, and especially furniture production, requires strategies aimed at enhancing competitiveness and securing a competitive advantage [6]. This necessitates the adoption of information technology in the furniture manufacturing industry to streamline business processes effectively and efficiently. To cater to the diverse information needs, the industry has implemented decision support systems (DSS), integral components of computer and knowledge-based systems that bolster decision-making processes within an industry [7, 8].
The implementation of a DSS provides a strategic advantage by serving organisational planning and augmenting the efficiency of making production decisions [9]. Within the furniture manufacturing industry, DSSs can be integrated across various divisions, including marketing, accounting, and production [10]. The objective of developing a DSS is to generate alternative decisions in the production process of goods, thereby necessitating a standard for quality assurance. It is posited that a system can only meet high-quality standards if it aligns with user needs [11].
From a broader perspective, the measurement of software quality can be approached from various angles, with a focus on user satisfaction being a notable consideration.
Therefore, the measure of software quality emerges as a critical factor in resolving user concerns or uncertainties related to the system [12]. These uncertainties frequently pertain to the speed of information reporting, order reception, and precision in calculating the usage of production raw materials in the context of a decision support system. A quality assessment of a decision support system is crucial in applying suitable evaluation standards. The purpose of such assessment is to measure usability, access convenience, and the level of maintenance from a user's perspective [13].
Standardised assessment parameters, such as those outlined in ISO 9126, provide a foundation for evaluating general software quality [14]. ISO 9126 delineates the quality of software products, models, and characteristics, thereby facilitating the determination of software product quality. It encompasses four software quality characteristics: functionality, reliability, usability, efficiency, maintainability, and portability [15]. This standard offers a means to evaluate software products in terms of their internal and external quality and the interrelationships with quality attributes.
Given the context of the challenges in the furniture industry, this research concentrates on applying the ISO 9126 model to assess the quality of a decision support system. The adaptability of ISO 9126 across the system enables a comprehensive evaluation of the software. Although the model presents several characteristics, the current study limits its focus to two, namely, functionality and reliability. Functionality pertains to the capacity of a decision support system to offer functions that meet user needs, while reliability refers to the system's ability to maintain a particular performance level.
Each characteristic of ISO 9126 is further divided into several sub-characteristics. The selection of two out of the six factors from the ISO 9126 model is informed by the goal to focus the system testing on internal components. Therefore, the objective of this study is to test a decision support system using the ISO 9126 characteristics of functionality and reliability.
Quality assessment is requisite for addressing user uncertainties about the decision support system. ISO 9126 proposes several statements leading to quality measurement related to functional appropriateness, data accuracy, interplay with other software, prevention of unauthorized access, amongst others. Consequently, the application of ISO 9126 seeks to mitigate potential perception bias in implementing a decision support system, which might not provide clear usage guidelines.
The outcomes of this study will manifest as an evaluation of user satisfaction concerning the application of a decision support system. Therefore, these test results may inform decision-making processes in the development of a decision support system. Moreover, the findings could potentially dispel user uncertainties about the system and enhance optimal business performance.
The research flow framework aims to describe the stages of applying ISO 9126. The research stages are shown in Figure 1 [16].
Figure 1 shows the research steps for applying the ISO 9126 model to a decision support system. The stages are divided into [17]:
(1) Literature Study. Obtaining the basic literature in solving problems and achieving the research objectives. The topics related to this research are the functionality and reliability characteristics of ISO 9126. Functionality was chosen as a measuring tool to evaluate a decision support system aligned with the needs, features, and capabilities. Meanwhile, reliability focuses on performing its intended function accurately and consistently over time. Thus, selecting these two characteristics is based on the current DSS condition, which needs to increase user trust and data integrity and thus reduce downtime.
(2) Data Collection. Obtain initial data to solve user problems in the decision support system. Data collection begins with conducting direct interviews with related parties. The interviews show that the user's problems are related to the user's doubts about the speed of reporting information, receiving orders, and the accuracy of calculating the use of production raw materials. The process of selecting informants for interviews from the primary users/actors who utilize a decision support system. Interviews on the use of the system on 36 users by asking 15 types of questions. Questions designed for the ISO 9126-related interview process focus on how a system can provide functionality and maintain a certain level of performance. The results of the interviews form the basis for designing user goals and weighting characteristics. Next, make observations by observing the activities that occur in the furniture manufacturing industry. Observations can show how the application runs and whether it is compatible with the existing functions by observing and obtaining the context of the use of the system and the target user. Apart from conducting interviews and observations, questionnaires were distributed to obtain user feedback.
Figure 1. Research stages
(3) Identification of General Process. Define the business processes that take place in the furniture manufacturing industry. The formulation of business processes aims to describe the problems that arise. In addition, this process also aims to map the use of modules in the system to overcome problems that arise.
(4) Define the Test Instrument. Evaluate the quality of use by assessing the extent to which the decision support system enables users to meet their needs. In addition, it is necessary to define user goals for the modules in the decision support system.
(5) Quality Metrics Identification. The design of quality metrics aims for quality measurement to have a clear context and become an effective measure in business processes. Identify quality metrics by defining each sub-characteristic of functionality and reliability of ISO 9126. Defining sub-characteristics by assessing business and system suitability.
(6) Evaluation Module. Define the modules in the decision support system and calculate user satisfaction with their use. The results of the evaluation come from distributing questionnaires to users.
(7) Respondent Assessment. Define the results of data analysis using ISO 9126 through User Acceptance Testing (UAT) and the user questionnaire. The UAT process involved 300 system users who were asked to test the features of each module.
(8) Summary. Describe the results of data analysis using ISO 9126 through User Acceptance Testing (UAT) and user questionnaires.
3.1 Identification of general process
Identifying business processes is one of the initial stages of the analysis process [18]. The business process is a series of business activities, including input initiation and transformation of information, in producing output. The description of business processes can take place from the initial stages of the ordering process by consumers to the final stage, namely, product delivery to consumers.
Figure 2 shows the conventional processes that occur in furniture production. Conventional processes in the manufacturing industry still use old methods with minimal use of technology. The business process currently running in the manufacturing industry begins with the marketing department conducting customer searches. The warehouse section makes travel documents to release goods from the warehouse. The travel document is submitted to the accounting department. The accounting section makes delivery letters. After the delivery letter is ready, the process of sending goods can run. Furniture delivery takes place to the customer's address. The accounting section makes financial reports (balance sheet, profit and loss, general ledger, etc.) [19].
3.2 Define test instrument
ISO 9126 defines software quality by identifying its key attributes [20]. The ISO 9126 quality factor in this study only focuses on the characteristics of functionality and reliability. The choice of ISO 9126 functionality characteristics is because testing focuses on the ability of features to meet user needs, ensure functionality is aligned with current conditions, and verify whether the decision support system meets the requirements during the development phase. From the reliability side of ISO 9126, testing focuses on minimizing unexpected failures/errors, reducing downtime, and the risk of data loss. The selection of these two characteristics is due to the critical nature of the user. Accurate and reliable functions are crucial for using a decision support system. The implication of choosing these two characteristics is that there will be an impact on the decision support system's user experience, performance, and long-term sustainability. The following describes the characteristics and sub-characteristics of ISO 9126 on decision support systems for the furniture manufacturing industry [21].
Table 1 shows the criteria for evaluating a decision support system using ISO 9126. Each sub-characteristic has several requirements to produce an effective and efficient decision support system. This question then became the basis for distributing questionnaires to 36 users of the decision support system. The type of attitude measurement scale on the answer choices of questionnaire respondents can use a Likert scale. The process continues by defining user goals for the system. User goals are descriptions of the final status that users want to achieve. User goals should refer to real-world end statuses and can be useful for generating insights. In the decision support system, the formulation of user goals is as follows:
(1) Minimizing errors when entering customer orders.
(2) Know the data collection of customers who buy goods.
(3) Knowing the calculation of the sale of goods to customers.
(4) Obtain master data from various divisions for processing at the beginning of the year.
(5) Obtain guidance on data collection on accounting groups.
(6) Obtain customer data collection guidelines, sales assistants, employees to agents.
(7) Obtain tax-related data collection guidelines.
(8) Know how to make work orders to be sent to the production department.
(9) Obtain guidelines for data collection for balance sheet reports, payables/receivables, transaction journals, sales, and fixed assets.
(10) Know how to monitor raw materials, leather fabrics, and finished goods data.
(11) Obtain local and foreign supplier information.
(12) Obtain guidelines for calculating fabric and component requirements for the production process.
(13) Know how to make purchase orders for imported and local goods.
(14) Obtain guidance on how to retrieve work order data and create detailed work orders.
(15) Obtain guidelines for reporting warehouse raw material stocks.
3.3 Quality metrics identification
Define quality metrics to control the quality of software products [22]. These metrics define the properties and complexity of the final product of the software. Quality metrics are the basis for evaluating the internal section (developers) on the suitability of technology implementation [23]. The quality metrics show in Table 2 and Table 3.
Figure 2. General process in manufacture industry
Table 1. Overview charateristics and sub-charateristics
|
Characteristics |
Sub-Characteristics |
Question |
|
Functionality |
Suitability |
Has the decision support system performed the required functions? |
|
Has the decision support system ever failed while in use? |
||
|
Does the decision support system fulfil all task functions according to user goals? |
||
|
Accurateness |
Has there ever been a failure when entering data into the decision support system with an intensity of five times? |
|
|
Has the decision support system produced good reporting that matches user needs? |
||
|
Interoperability |
Can the decision support system interact with other systems? |
|
|
Security |
Does the decision support system meet the criteria for preventing unauthorized access? |
|
|
Does the decision support system meet the standards and requirements following applicable regulations? |
||
|
Reliability |
Maturity |
Can the decision support system avoid failures or errors in the software? |
|
Have hardware errors in the decision support system implementation been minimized? |
||
|
Fault Tolerance |
Can the decision support system manage its performance when unexpected errors occur? |
|
|
Is the decision support system capable of managing performance levels when software and hardware problems arise? |
||
|
Recoverability |
Is the decision support system able to return to normal work even though complex failure problems arise? |
|
|
Is the decision support system capable of backing up data in the event of a failure? |
||
|
Reliability Compliance |
Does the decision support system comply with the reliability standard of the software? |
Table 2. Functionality quality metrics
|
Sub-Characteristics |
Quality Metrics |
|
Suitability |
Provides the right set of functions and purposes. |
|
Accurateness |
Provides expected results or specified goals, such as create, update, delete, and search bar buttons. |
|
Interoperability |
Restrict access from other software and provide usage guide content. |
|
Security |
Maintain the confidentiality of information, including authentication processes, login procedures, display of sensitive data, password protection and user privileges. |
|
Compliance |
Maintain the limitations of the information provided. |
Table 3. Reliability quality metrics
|
Sub-Characteristics |
Quality Metrics |
|
Maturity |
In terms of software failure frequency and error-free functionality. |
|
Fault Tolerance |
Responding to invalid input and ability to maintain performance in a software error. |
|
Recoverability |
Resumes work immediately after the failure, quickly recover from failure, and alert users to take action after errors are recognized. |
|
Reliability Compliance |
Compliance with software development standards. |
3.4 Module evaluation
Evaluation of the module aims to map the list of pages from the decision support system [24]. Within the scope of implementing a decision support system, there are several important requirements outlined in several modules, namely:
(1) Marketing
(2) Accounting
(3) Production
The user interface implementation of each module is shown in Figure 3 and Figure 4.
Figure 3 shows the login module page for each user access in the marketing, accounting, and production sections. The marketing department uses this form to enter the account. Users can enter a registered account name and password. After successfully logging in to the account, the user can see the main page view. Figure 4 shows the appearance of the main page of the decision support system in the accounting section. The finance section uses this form to select the financial menu to view. Users can access the main page, transactions, marketing, general ledger, reports, fixed assets, custom/other settings, and exit the system.
Furthermore, to help evaluate a decision support system, you can use the User Acceptance Test (UAT). UAT is the process of verifying the functionality of a solution for its users [25, 26]. UAT engages the user to know what the system does and what the user gets. UAT is included in black-box testing where the user is not interested in the internal/system coding but compares system functions with requirements. Due to page limitations for the details of the entire process, only a few modules are displayed.
Table 4 shows the results of user acceptance testing on several modules. The challenges or limitations during the UAT process are related to the limited scope of testing and little user expertise. The range of testing is limited because it usually only focuses on common and essential user scenarios. Thus, it may not cover all possible actual usage scenarios. Little user expertise due to insufficient technical knowledge to carry out in-depth testing results in problems not being detected. Limitations of the UAT process are overcome by involving users from the start of developing a decision support system, from exploring requirements to the functional testing stage.
3.5 Respondent assessment
After collecting the questionnaire results, the next step is to make a recapitulation table for calculating the eligibility percentage from the results of the respondents' assessment. Questionnaires were distributed to 300 employees of the manufacturing industry from various positions. Table 5 shows the results of the respondents' satisfaction assessment.
Table 5 shows the results of a good assessment for the overall characteristics of ISO 9126 for the decision support system.
Figure 3. Get user data
Figure 4. Home module
Table 4. Evaluate module
|
Test Case |
Test Steps |
Expected Result |
Actual Result |
|
Delivery Address |
|||
|
Add new data |
1. Select the “Delivery Address” sub-menu 2. Select “New” |
All fields will be opened |
All fields have been opened successfully |
|
Save new data |
1. Select the “Delivery Address” sub-menu 2. Select “New” 3. Fill in the company name 4. Add an address 5. Click “Save” |
New data added successfully |
Sending address data added successfully |
|
Change company name |
1. Select the “Delivery Address” sub-menu 2. Select "Edit" 3. Change the company name 4. Click “Save” |
The company name will change |
Company name successfully changed |
|
Change address |
1. Select the “Delivery Address” sub-menu 2. Select "Edit" 3. Change the address 4. Click “Save” |
The company address will change |
Company address changed successfully |
Table 5. Respondent assessment result
|
Characteristics |
Score |
Criteria |
|
Suitability |
4.17 |
Good |
|
Accurateness |
4.28 |
Good |
|
Interoperability |
4.36 |
Good |
|
Security |
4.17 |
Good |
|
Maturity |
4.17 |
Good |
|
Fault Tolerance |
4.19 |
Good |
|
Recoverability |
4.25 |
Good |
|
Reliability Compliance |
4.00 |
Good |
Technological developments can make data processing easier and produce more accurate information, especially in manufacturing. This study yielded good quality findings on decision support systems in terms of suitability, accuracy, interoperability, security, maturity, fault tolerance, recoverability, and reliability compliance. The furniture manufacturing industry has implemented a decision support system to accommodate every information need. However, there are various kinds of doubts about the development of a decision support system. Various user doubts about the decision support system are related to the speed of reporting information, receiving orders, to the accuracy of calculating the use of production raw materials. Quality evaluation in this study uses the help of ISO 9126. Each ISO 9126 characteristic is divided into several sub-characteristics. Functionality characteristics have sub-characteristics such as suitability, accuracy, interoperability, and security. The reliability category has maturity, fault tolerance, recoverability, and reliability compliance sub-characteristics. The first step is to formulate conventional business processes in the industry. To assist evaluation, the formulation of criteria for questions and statements became the basis for distributing questionnaires to 300 users of the decision support system. Formulation of questions based on ISO 9126 sub-characteristics.
Next, the process continues by defining user goals for the system. User goals are the final expectations users want to achieve in marketing, accounting and production. Quality metrics results show compatibility between business processes and alignment with technology. Several modules prove the suitability of the process. Furthermore, to help evaluate a decision support system, you can use the User Acceptance Test (UAT). UAT engages the user to know what the system does and what the user gets. After collecting the questionnaire results, the next step is to make a recapitulation table for calculating the eligibility percentage from the results of the respondents' assessment. The results of the percentage of respondents indicate that the level of user satisfaction achieves a good score. Thus, applying a decision support system can help overcome user doubts and problems experienced in conventional business processes. A high level of user satisfaction can have significant practical benefits for the manufacturing industry. Practical benefits in the manufacturing industry are establishing higher customer loyalty, meeting expectations, and providing added value products, fewer complaints and product returns, and better product innovation.
Overall, high levels of customer satisfaction have had a broad positive impact on the manufacturing industry. Not only creating stronger relationships with customers but also influencing the industry's financial performance, reputation, and long-term sustainability. This research has limited resources, namely time to conduct a comprehensive quality evaluation. Therefore, this study only focuses on the characteristics of functionality and reliability. Therefore, for future research, further improvements can be made to ISO 9126 so that it significantly impacts the overall quality of the decision support system. The characteristics in question are related to usability, efficiency, maintainability, portability, effectiveness, and safety.
This research was supported by the Doctoral program of Information Systems, School of Postgraduate Studies, Diponegoro University and Furniture Industry.
[1] Javaid, M., Haleem, A., Vaishya, R., Bahl, S., Suman, R., Vaish, A. (2020). Industry 4.0 technologies and their applications in fighting COVID-19 pandemic. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 14(4): 419-422. https://doi.org/10.1016/j.dsx.2020.04.032
[2] Tallon, P.P., Queiroz, M., Coltman, T., Sharma, R. (2019). Information technology and the search for organizational agility: A systematic review with future research possibilities. The Journal of Strategic Information Systems, 28(2): 218-237. https://doi.org/10.1016/j.jsis.2018.12.002
[3] Li, C.Z., Zhao, Y., Xiao, B., Yu, B., Tam, V.W., Chen, Z., Ya, Y. (2020). Research trend of the application of information technologies in construction and demolition waste management. Journal of Cleaner Production, 263: 121458. https://doi.org/10.1016/j.jclepro.2020.121458
[4] Wamba-Taguimdje, S.L., Fosso Wamba, S., Kala Kamdjoug, J.R., Tchatchouang Wanko, C.E. (2020). Influence of artificial intelligence (AI) on firm performance: The business value of AI-based transformation projects. Business Process Management Journal, 26(7): 1893-1924. https://doi.org/10.1108/BPMJ-10-2019-0411
[5] Winkelhaus, S., Grosse, E.H. (2020). Logistics 4.0: A systematic review towards a new logistics system. International Journal of Production Research, 58(1): 18-43. https://doi.org/10.1080/00207543.2019.1612964
[6] Azeem, M., Ahmed, M., Haider, S., Sajjad, M. (2021). Expanding competitive advantage through organizational culture, knowledge sharing and organizational innovation. Technology in Society, 66: 101635. https://doi.org/10.1016/j.techsoc.2021.101635
[7] Irawan, Y. (2020). Decision support system for employee bonus determination with web-based simple additive weighting (SAW) method in PT. Mayatama Solusindo. Journal of Applied Engineering and Technological Science (JAETS), 2(1): 7-13. https://doi.org/10.37385/jaets.v2i1.162
[8] Patalay, S., Bandlamudi, M.R. (2021). Decision support system for stock portfolio selection using artificial intelligence and machine learning. Ingénierie des Systèmes d’Information, 26(1): 87-93. https://doi.org/10.18280/isi.260109
[9] Zhai, Z., Martínez, J.F., Beltran, V., Martínez, N.L. (2020). Decision support systems for agriculture 4.0: Survey and challenges. Computers and Electronics in Agriculture, 170: 105256. https://doi.org/10.1016/j.compag.2020.105256
[10] Yan, M.R., Tran-Danh, N., Hong, L.Y. (2019). Knowledge-based decision support system for improving e-business innovations and dynamic capability of IT project management. Knowledge Management Research & Practice, 17(2): 125-136, https://doi.org/10.1080/14778238.2019.1601507
[11] Correia, A.P., Liu, C., Xu, F. (2020). Evaluating videoconferencing systems for the quality of the educational experience. Distance Education, 41(4): 429-452. https://doi.org/10.1080/01587919.2020.1821607
[12] Thota, M.K., Shajin, F.H., Rajesh, P. (2020). Survey on software defect prediction techniques. International Journal of Applied Science and Engineering, 17(4): 331-344. https://doi.org/10.6703/IJASE.202012_17(4).331
[13] Martínez-Sala, A.M., Monserrat-Gauchi, J., Alemany-Martinez, D. (2020). User usable experience: A three-dimensional approach on usability in tourism websites and a model for its evaluation. Tourism Management Perspectives, 33: 100579. https://doi.org/10.1016/j.tmp.2019.100579
[14] Yadav, S., Kishan, B. (2020). Analysis and assessment of existing software quality models to predict the reliability of component-based software. International Journal of Emerging Trends in Engineering Research, 8(6). https://doi.org/10.30534/ijeter/2020/96862020
[15] Suryanti, S., Sutaji, D., Nusantara, T. (2023). Online learning quality measurement application for higher education: Development and analysis using ISO 9126. KnE Social Sciences, 94-107. https://doi.org/10.18502/kss.v8i9.13323
[16] Gokul, Y., Gopal, M. (2017). An authoritative method using fuzzy logic to evaluate maintainability index and utilizability of software. Advances in Modelling and Analysis B, 60(3): 566-580. https://doi.org/10.18280/ama_b.600304
[17] Firdaus, B., ST, A.M., Banowosari, L.Y., Kom, S. (2021). Quality analysis of payroll information system based on ISO 9126 In PT Karya Prima Usahatama. International Journal of Research Publications, 71(1): 11. https://doi.org/10.47119/IJRP100711220211754
[18] Viriyasitavat, W., Hoonsopon, D. (2019). Blockchain characteristics and consensus in modern business processes. Journal of Industrial Information Integration, 13: 32-39. https://doi.org/10.1016/j.jii.2018.07.004
[19] Andry, J.F., Hadiyanto, H., Kadarisman, V.G.S. (2021). Comparison elicitation techniques in furniture manufacture requirement. In E3S Web of Conferences. EDP Sciences, 317: 05022. https://doi.org/10.1051/e3sconf/202131705022
[20] Kassie, N.B., Singh, J. (2020). A study on software quality factors and metrics to enhance software quality assurance. International Journal of Productivity and Quality Management, 29(1): 24-44. https://doi.org/10.1504/IJPQM.2020.104547
[21] Andry, J.F., Gunawan, V. (2023). Critical factors of supply chain based on structural equation modelling for Industry 4.0. Journal Européen des Systèmes Automatisés, 56(2): 187-194. https://doi.org/10.18280/jesa.560202
[22] Santos, G., Murmura, F., Bravi, L. (2019). Developing a model of vendor rating to manage quality in the supply chain. International Journal of Quality and Service Sciences, 11(1): 34-52. https://doi.org/10.1108/IJQSS-06-2017-0058
[23] Wagire, A.A., Joshi, R., Rathore, A.P.S., Jain, R. (2021). Development of maturity model for assessing the implementation of Industry 4.0: Learning from theory and practice. Production Planning & Control, 32(8): 603-622. https://doi.org/10.1080/09537287.2020.1744763
[24] Alamanos, A., Rolston, A., Papaioannou, G. (2021). Development of a decision support system for sustainable environmental management and stakeholder engagement. Hydrology, 8(1): 40. https://doi.org/10.3390/hydrology8010040
[25] Soko, I.P. (2022). The Evaluation of web-based and android face-to-face tutorial applications quality using the user acceptance testing (UAT) method. Journal of World Science, 1(8): 590-595. https://doi.org/10.58344/jws.v1i8.76
[26] Pillai, N.S.R., Hemamalini, R.R. (2022). Hybrid user acceptance test procedure to improve the software quality. International Arab Journal of Information Technology, 19(6): 956-964. https://doi.org/10.34028/iajit/19/6/14
