Web Usability Testing on Diabetic Retinopathy Consultation System

Web Usability Testing on Diabetic Retinopathy Consultation System

Nurul Najihah A’bas Sarni Suhaila RahimMohamad Lutfi Dolhalit Wan Sazli Nasaruddin Saifudin Nazreen Abdullasim Shahril Parumo Raja Norliza Raja Omar Siti Zakiah Md Khair Khavigpriyaa Kalaichelvam Syazwan Izzat Noor Izhar

Fakulti Teknologi Maklumat Dan Komunikasi, Universiti Teknikal Malaysia Melaka (UTeM), Durian Tunggal, Melaka 76100, Malaysia

Department of Ophthalmology, Hospital Melaka, Jalan Mufti Haji Khalil, Melaka 75400, Malaysia

Corresponding Author Email: 
sarni@utem.edu.my
Page: 
255-264
|
DOI: 
https://doi.org/10.18280/isi.260302
Received: 
13 March 2021
|
Accepted: 
26 May 2021
|
Published: 
30 June 2021
| Citation

OPEN ACCESS

Abstract: 

This article presents an investigation on the usability testing of Diabetic Retinopathy Consultation System (DRCS) development. DRCS is an interactive medical web-based consultation system for diabetic retinopathy screening with the features of image processing and image editing. The expected outcome of this paper is a comprehensive analysis of the web usability towards the development of a novel consultation system for diabetic retinopathy screening. Seven important usability evaluation components were included in this testing in order to investigate the user engagement and satisfaction of the proposed medical consultation system. Online questionnaires were distributed as a method to collect user testing outputs. A total of 27 respondents from the health clinics were involved in this survey. According to the outcomes of the online survey, the majority of respondents are satisfied with the outcomes of the DRCS. In conclusion, the online findings depict that the DRCS is highly functional, as required by the target users to assist consultation among medical doctors on diabetic retinopathy screening. This system is expected to contribute to diabetic retinopathy screening management and produce a great usability to the users.

Keywords: 

consultation, diabetic retinopathy, eye screening, web development, image editing, image processing, usability

1. Introduction

Diabetic Retinopathy (DR) is a retinal vasculopathy condition caused by long-term diabetes mellitus that can lead to blindness. Diabetic retinopathy is the most prevalent ocular consequence of diabetes and a leading cause of adult vision loss [1]. Diabetic retinopathy screening, which leads to early detection of the illness and prompt treatment, can help persons with diabetes avoid visual loss. This disease is caused by diabetes that involves impairment to the blood vessels in the back of the eye [2]. Diabetic retinopathy is a collection of retinal lesions occurring in persons who have had diabetes for a long period of time [3]. Hence, detailed DR screening is necessary for both early diagnosis and early treatment.

In order to help retinal screeners, identify retinal images effectively, a thorough retinal examination is needed. The difficulty in diabetic retinopathy screening occurs when inexperienced and junior medical officers in health clinics have difficulty detecting and diagnosing diabetic retinopathy and may refer to senior medical officers or ophthalmologists in hospital. Thus, the primary objective of this research is to collect data of system feedback from the target users, to analyze the results of the target user’s feedbacks on web-based clinical decision support system for diabetic retinopathy screening and to report on the study's findings. Therefore, this study able to identify the usability for each feature proposed in the web-based system, including the features able to solve the existing limitations in diabetic retinopathy screening and meet the target users’ requirements.

In this article, the key knowledge concerning DR, web-based consultation components in existing work will be mentioned in the following section, a brief overview on research materials and method used will be shown in the section three, which includes image processing and image editing features, and the section four will discuss the findings of this research.

2. Literature Review

2.1 Diabetic retinopathy

Diabetic retinopathy is a long-term microvascular complication with a poor prognosis. Nearly every patient with chronic diabetes mellitus develops it to some degree. DR can lead loss of vision and even blindness [4]. Therefore, it is important to have a consultation platform for medical doctors to determine and classify the retinopathy stages and DR management based on the patient’s eye fundus images. The early signs of DR which appear on the surface of the retina are dark lesions such as microaneurysms (MAs), haemorrhages (HEMs), and bright lesions (BLs) such as exudates [5]. Microaneurysms, retinal haemorrhages, hard exudates, cotton wool patches, irregular new vasculature, and venous beadings are some of the retinopathy signs that have been identified as shown in Figure 1 [6].

2.2 Diabetic retinopathy screening

The ultimate focus of diabetic retinopathy screening is to determine whether patients need to be referred for further treatment in order to avoid blindness [7]. In addition, DR screening is also a way to reduce diabetes risk, such as visual morbidity from diabetic retinopathy, by encouraging early detection and treatment of visual retinopathy through population screening [6]. One major concern is that diabetic eye disease does not damage sight until it has evolved to a stage where it is irreversible [7]. The use of laser treatment may aid save an individual's vision, but only if it is diagnosed early. This emphasizes the need of routine screening, which can assist diabetic patients diagnose diabetic retinopathy at an early stage. Furthermore, earlier detection of any retinopathy symptoms may allow for improvements in blood pressure or blood glucose management to decrease the progression of the disease.

Figure 1. Diabetic retinopathy features [6]

2.3 Diabetic Retinopathy Detection System

In the literature, numerous systems for detecting and diagnosing DR have been published. Some of the new advancements are focused on and provide ways for recognizing specific DR characteristics.

The DR detection system is one of the available DR systems that aims to help in DR detection. In 2016, Rahim et al. [8] focused primarily on detecting microaneurysms from a color fundus image. Furthermore, this study employs pre-processing techniques such as green channel extraction, histogram equalization, and morphological processes. Their results are 96% and 92% respectively, indicating the DR detection system's sensitivity and specificity [8].

Another automated detection of diabetic retinopathy using radial base function (RBF) neural network classifier retinal image, which is automatically identified by the system as Non-DR or DR [9]. Pre-processing, image enhancement, ROI extraction, and ocular characteristics such as blood vessels, exudates, and microaneurysms were all used in this A-IFS histone-based segmentation method investigation [9]. Furthermore, using the RBF classifier, the study was able to identify retinal images as DR or No DR. This system has the potential to improve the performance of classifiers and the system's accuracy in the future.

Besides that, Bellemo et al. [10] present a study on attributable and eyesight diabetic retinopathy in Africa utilizing artificial intelligence (AI) for screening with deep learning. This study aims to detect and reduce diabetes-related blindness in order to meet the Sustainable Development Goals by 2030 [10]. In demographically diabetic retinopathy screening, an artificial intelligence (AI) method shows clinically significant performance in diagnosing referable diabetic retinopathy, vision-threatening DR, and diabetic macular oedema [10]. Even though the model is taught in a different demographic, this highlights the potential usage and application of AI is being used in a low-resource African community to lower the prevalence of irreversible blindness. In terms of referable diabetic retinopathy prevalence diagnosis and systemic risk factor linkages, the AI model and human evaluators produced identical outcomes.

Another existing diabetic retinopathy system can be found in Gargeya and Leng's [11] work, which built an automated DR system. This study, on the other hand, leverages data-driven deep learning theory to build robust diagnostic solutions to automate DR screening. The algorithm analyzed color fundus images and categorized them as safe (no retinopathy) or dangerous (with DRA comprehensive data-driven AI-based classification approach may be used to analyze fundus photos obtained from diabetic patients and classify, with great precision, which cases should be referred to an ophthalmologist for further analysis and treatment [11]. The total experiment indicates the usability of the presented model for successful DR diagnosis, aside from its desirable attributes, like as histogram-based classification and a deep learning-based model.

Shankar et al. [12] also provide a synergistic deep learning model for automatic identification and categorization of fundus diabetic retinopathy images. Deep learning was utilized in the automatic identification and classification model of the DR fundus image. Pre-processing, segmentation, and classification were all used in this image processing method. Due to its outstanding characteristics such as histogram-based segmentation and the most recent deep learning model, the total experimentation supports the significance of the proposed model for effective diagnosis of DR [12].

The Xie et al. [13] research is another AI-related research for teleophthalmology-based diabetic retinopathy screening. Furthermore, this research used the human referral rate based on actual data and Singapore referral guidelines, that may involve some referral bias and hence modify the high specificity of this evaluation model.

A diabetic retinopathy screening telemedicine system is also existing, which includes a license-free web-based DRS system intended to make image collecting, sharing, and examination easier [14]. This prepared work focuses on the development of telemedicine information systems for the treatment of psoriasis using neural network analyses of treatment details and patient effects. This intelligent system comprises pre-diagnosis, image storage in a database, and merely aggregates information to make certain predictions, which this system established to make certain features predictions.

HEMAN, an IoT-based e-health care system for remote telemedicine, is one more alike telehealth technology that make it significant development and engaged effectively with patients and clinicians [15]. This online health screening system collects data for storage in a database and generates a screening health report. The device is tested with eight medical parameters and live streaming, and the results are satisfactory.

Meanwhile, Telecare Medical Information Systems (TMIS) is another telemedicine technology that has advanced significantly and successfully communicated with patients and doctors [16]. This system outlines secure patient-doctor communication protocols, as well as the confidentiality of important data. Data protection, patient anonymity, integrity of the data, preliminary examination, and authentication process are all guaranteed by the protocols, which are constructed around a symmetrical key.

The aim of this research review is to assist the reader in identifying the current DR system. The system that has been proposed; DRCS, provides a novel concept for integrating image visualization and photo editing in a web-based consultation system. However, there is no credible evidence that has led to a conclusion. This could imply filling in the gaps in previous research. Table 1 indicates that a consultation system for DR has not been established where the existing systems lack of features such as image processing and image editing in a web-based version. Automated detection of DR and telemedicine or telecare technologies were proposed in the DR research or study. As a result, this research will be a novel investigation into integrating image visualization and photo editing as a core in a web-based DR system.

2.4 Medical consultation system

One of the most important aspects in a healthcare organization is the consultation system. There have been various systems documented in the literature that provide consultation in healthcare field.

In 2018, Pronsawatchai et al. [17] revealed that they analyzed and developed the new Rama Health Electronic Consulting (RHEC) system, which focused on two-way communication to promote coordination and collaboration between the specialized care team. According to previous study by Liddy et al. [18] they are working on the development and implementation of eConsult, which allows images to be kept and evaluated by the specialist at a later time.

Besides that, Boegl et al. [19] introduced the fuzzy knowledge representation framework of the medical computer consultation system MedFrame/CADIAG-IV, which included knowledge acquisition, knowledge representation, and knowledge-based systems. In their study, they described the knowledge representation and acquisition techniques that help medical specialists add, revise, and update their information in MedFrame/CADIAG-IV.

In addition, based on Banks et al. [20] research, they focused on eConsult, an online platform built by the Hurley group and currently supplied by EMIS Health. The objective of the eConsult is to provide patients with online access to advice and care through their GP practice's website [20]. Patients can utilize an eConsult sign analyzer to access self-help materials and videos on common ailments, as well as pharmaceutical guidance and a connection to NHS 111, and perform administrative tasks like requesting a repeat prescription or submitting an e-consultation [20].

Overall, the medical consultation system that have produced, is no credible evidence that has led to a conclusion. This could imply filling in the gaps in previous research. Table 1 indicates that a consultation system for DR has not been established where the existing systems lack of features such as image processing and image editing in a web-based version.

2.5 Web usability testing

Understanding of the term usability testing is a key tool for making goods more useable. Testing of usability is a method of assuring that a software can effectively perform one or more defined functions. According to Black [21] in other terms, it relates to determining how simple it is for users to learn how to utilize a product. Software companies can acquire data on their own product usability by doing usability testing. Usability testing is not officially used into a single component or system validation plan, despite the fact that it is probably vital as automated and manual testing of basic functionality [21]. In general, software that performs a set of functions is insufficient. Merely as there is a mechanism to conduct functional activities may not ensure that the interaction with the software is easy or simple to use.

According to Osterbauer et al. [22] usability can be described as the easiness with which a system may be learnt or used, its safety, efficacy, and efficiency, including the attitude of its users toward it. Rather than merely evaluating prototypes or design documents, the most efficient way to assure usefulness is treating human factors as an element to customize. Usability is effective when a strategy is developed those results in major usability advantages.

2.5.1 Functionality testing

The effort required to test an application's functionality is determined by the application's size and complexity, regardless of the environment in which it runs. According to Sneed [23], the effort necessary to test an application's operation is dictated by the level and difficulty of the product, regardless of the environment in which it is executing. In addition, according to study by Sneed [24], the work necessary to analyze a software's suitability through its surroundings is prompted by the environment’s difficulty. The difficulty is determined by the amount of interactions and interfaces that exist between the application and its surroundings.

Table 1. Summary of DR literature

Authors

Features

Automated System

Telemedicine

Consultation

Bellemo et al.

Deep learning

Yes

-

-

Chanchal et al.

IoT-based system

Yes

Yes

-

Gargeya and Leng

Data-driven deep learning

Yes

-

-

Kamble and Kokate

Radial base function neural network

Yes

-

-

Kondakov and Kulik

Web-based system, neural network

Yes

Yes

-

Rahim et al.

Image processing, machine learning

Yes

-

-

Rezaeibagha and Mu

Information system, symmetrical key

Yes

Yes

-

Shankar et al.

Synergistic deep learning

Yes

-

-

Tumpa et al.

Mobile system

Yes

Yes

-

Xie et al.

Deep learning

Yes

-

-

2.5.2 Usefulness testing

The purpose of usefulness testing is to determine how valuable specific features, functionality, content, and data are to users [25]. In 2021, Ambalov [26] perceives usefulness as a basic notion representing system efficacy in enhancing work performance. The concept of perceived usability namely, external advantages is acknowledged in information technology research as a key predictor of attitudes and intentions of the users to support and use technology, as well as, conclusively, the behavior itself.

2.5.3 Ease of use testing

The Technology Acceptance Model (TAM), established by Davis in 1989, is the most extensively utilized model, according to McCloskey [27]. TAM was created primarily to assess the factors that influence computer usage. According to the model, perceived ease of use influences attitudes toward use, which in turn influences behavioral intentions, which in turn influences actual usage. Furthermore, perceived ease of use influences usefulness in the sense that a system must be simple to use for people to use it and therefore appreciate its use.

2.5.4 Ease of learning testing

The ease of learning testing enables the web system's learnability to be evaluated. Furthermore, according to Lauesen and Younessi [28], ease of learning is evaluated in terms of how the system must be simple to comprehend for both beginners and experts with similar systems to understand. As stated in McKenzie and Edgell's [29] research, clinicians able to determine how often many patients have learned and understood diagnostic tests or processes by asking them to explain the diagnostic tests or processes they recently discussed with their doctor or nurse. Tests that generate learnability data can help organizations gain a better understanding of their testing.

2.5.5 Satisfactory testing

The satisfactory testing is the comfort and acceptability of use [30]. Rinder et al. [31] outline some of the perspectives and methodologies of website usability testing pundits, including an overview of Jakob Nielsen's concepts. They recognized the value of Nielsen's online usability testing methodologies and techniques but felt that a one-size-fits-all approach is short-sighted and fails to account for something as amorphous as website design.

3. Development

The system architecture of the DRCS web-system is built with combination of programming languages and scripting such as HTML, PHP, JavaScript, jQuery, CSS, and SQL for the database management. This web-system is coded in the cPanel, and it is hosted under Linux web server to allow this web-system can be available online.

There are four core functions included in the DRCS such as Create, Read, Update and Delete (CRUD) function, image processing, image editing and forum. The CRUD function is used to handle the patient’s DR case record. As for the image processing feature used during magnifying the fundus image and change the fundus image into grey scale filter. This feature needed to view the DR features in the fundus image especially the complex feature for instance microaneurysms.

In addition, image editing used during the specialist review the DR case record which include the fundus images that contains DR features. In annotation features include editing tools such as box, circle, text, arrow, and draw would be able to help specialists in responding to the DR fundus image by detecting the critical features of DR as shown in Figure 2. Moreover, this system applies many-to-many conversation where the primary health cares able to communication with the specialists through this online platform where discussion on the matter of DR cases able to be done in mass discussion.

Figure 2. Illustration of applying annotation feature of image editing on the fundus image (fundus image taken from the [32])

4. Result and Discussion

Web usability test were conducted to analyze the system’s usefulness. The target user, primary health care doctors from health clinics in Melaka, Malaysia, compassed this test. The testing focuses on the usability of the web-functionalities. The detailed explanation on the development of DRCS, including the analysis, design and implementation phases are presented previously [33]. Table 2 summarizes the web usability testing conducted. Additionally, these findings are vital for this study's goal of determining the features' use for DR screening consultation.

Table 2. Summary of the survey conducted

Date

5/8/2020

Type of survey

Online survey via Google Form

No. of respondents

27 respondents

No. of workplace

Health Clinics, Melaka, Malaysia

Type of respondents

Primary Health Care

No. of survey parts

7 components:

  • Demographic
  • Functionality
  • Usefulness
  • Ease of use
  • Ease of learning
  • Satisfactory
  • Suggestion for improvement

4.1 Demographic

The demographic data of the responses component of the survey intends to include an outline of the responses' known knowledge such as gender, age, former place of work, and years of work experience. This information can be used to analyze the consultation web-usefulness. The responses are mainly women between the ages of 30 and 40, according to the results reported in Table 3. The finding suggests that the responders are currently employed at a Health Clinic and have 6-8 years of medical experience.

Table 3. Demographic data on usability testing

 

Primary Health Care

Gender

  • Male
  • Female

14

13

27

Age

  • Less than 20 years old
  • 20-30 years old
  • 30-40 years old
  • 40-50 years old
  • More than 50 years old

 

0

2

23

2

0

27

Current Workplace

  • Hospital
  • Health Clinic

0

27

27

Working Years

  • Less than 3 years
  • 3-5 years
  • 6-8 years
  • 8-10 years
  • More than 10 years

0

10

13

3

1

27

4.2 Functionality

The functionality testing includes the DR cases as well as image processing. The outcomes of the testing are shown in Table 4. According to the gathered data, the majority of respondents effectively use the system's functionalities, which include the DR cases view, image filtering tool capabilities, and image magnification facilities.

Table 4. Functionality testing

Successfully submit DR case details

Successfully view diabetic retinopathy case details submitted

Successfully delete diabetic retinopathy case details

Yes

No

27

0

Yes

No

27

0

Yes

No

22

5

Able to view a list of diabetic retinopathy case details submitted by user

Successfully upload fundus images

Able to view fundus images

Yes

No

27

0

Yes

No

27

0

Yes

No

27

0

Grayscale feature is functioning well

Magnifier function is working fine

Yes

No

27

0

Yes

No

25

2

4.3 Usefulness

The usefulness aspect is evaluated based on the functions available in the DR cases and image processing. The outcomes of the functionality testing are shown in Figure 3. On the basis of the total outcome, most of the respondents satisfied with the usefulness of the system in terms of communicating with the specialist regarding the DR case, the ease of learning to use the system, the usefulness of the grayscale feature on the image uploaded, and the effectiveness of the magnifier feature to enlarge the image uploaded.

Figure 3. Usefulness testing

Figure 4. Ease of use testing

4.4 Ease of use

The system's ease of use is analyzed according to the system's ease of use. The outcomes of the ease of use tests are shown in Figure 4. According to the results obtained, the majority of responses strongly agree on the organization of onscreen information, the system is simple to use, less interactions needed, gives error message, and ease in finding information in the web-system.

4.5 Ease of learning

This section comprises the evaluation outcomes for the system's ease of learning. Figure 5 present the outcomes of this web usability test. As a whole, this testing revealed the most respondents agree with the ease of learning of the system content and features such as ease in learning to use the system, quickly became skillful using the web-system, and interaction the system is flexible.

4.6 Satisfactory

The system's satisfaction was evaluated by how well it met the needs of the target users. From this section, respondents found out the criteria needed are presented well such as the system acts as the learning medium, simplified the learning process, enhance the effectiveness in DR signs consultation, include all functions and capabilities for DR signs consultation and overall satisfaction level with the web-system. Figure 6 shows the system's satisfactory results. Overall, the findings of this testing reveal that the respondents are satisfied and fulfilled with the web system's performance.

4.7 Discussion

The usability testing includes seven components for DRCS testing, including demographic, functionality, usefulness, ease of use, ease of learning, satisfaction, and suggestions for improvement from respondents.

Figure 5. Ease of learning testing

Figure 6. Satisfactory result

In total, 27 responders from health clinics in Melaka, Malaysia, took part in the system's testing. Respondents made some suggestions for improving the usability of the web-system. For example, it is recommended to include annotations and picture editing in primary health care site in order to allow the doctors to highlight areas that are confusing. Besides that, using dark background for user’s view would ease of use under dark condition, besides applying more colors to the web-system also been suggested by the respondents. In addition, adding an appointment date for patient that need to be seen at the eye clinic also has been suggested. Moreover, the respondents also suggest to give option to choose for submitting only one eye referral for patients who have only one eye. It is also suggested to add live feed video or audio in order for the specialist able to do live consultation and give review on the referral clearer.

From the usability evaluation conducted for DRCS, it is clearly demonstrated that the medical system requires related and important information and features, straight-forward system flow, ease of use, system completion and focusing on the particular disease practice. For instance, the medical doctors highlighted only two main features of image editing to be incorporated in DRCS, namely grayscale image conversion and magnifier tool, which are highly required for the eye fundus image processing, although there are various image editing and image processing features that can be included. Moreover, the system is following the specific flow of the practice of DR screening; starting from the image acquisition, patients’ management, image pre-processing, detection, diagnosis and disease management. It shows that the medical system is more focusing on the reliability and validity of the result generated, as in medical, the correct detection and diagnosis of the disease is very crucial.

5. Conclusions

This report outlines the successful design, and implementation of a DR consultation system ability to integrate image processing and image editing on a web-based platform. Hence, this research of consultation system provides a novel DR system in a web-based method for the consultation on the DR cases between the specialists in hospital and primary health cares in health clinics. Despite the fact that the developed DRCS promotes the channel of communication between the specialists and primary health cares.

This article covers the DRCS testing phase. The effort aided in the sectors of medicine and education. Furthermore, by using this application, it raises users' familiarity with computer technologies that is useful. The findings of this study will get a significant impact on the fields of medicine and education by increasing the efficiency of medical specialists in treating vision diseases. This initiative would serve as a benchmark or reference for several eye illnesses including radiography, corneal disease, dermatitis, and hypertensive retinopathy.

Acknowledgment

This research is part of Master of Science in Information and Communication Technology, presently being carried out at the Faculty of Information and Communication Technology, Universiti Teknikal Malaysia Melaka (UTeM). UTeM funded this research entirely through Short Term High Impact Research Grant (PJP/2020/FTMK/HI17/S01715). We are thankful to all primary health cares who took part in the final survey and testing phase, and we would like to express our thoughtful appreciation to them.

  References

[1] Felman, A. (2020). What to know about diabetic retinopathy. Tratto da Medical News Today. https://www.medicalnewstoday.com/articles/183417.

[2] National Health Service. (2020). Diabetic retinopathy - NHS. Tratto da NHS: https://www.nhs.uk/conditions/diabetic-retinopathy/.

[3] Solomon, S.D., Chew, E., Duh, E.J., Sobrin, L., Sun, J.K., VanderBeek, B.L., Wykoff, C.C., Gardner, T.W. (2017). Diabetic retinopathy: A position statement by the American Diabetes Association. Diabetes Care, 40(3): 412-418. https://doi.org/10.2337/dc16-2641

[4] Magliah, S.F., Bardisi, W., Al Attah, M., Khorsheed, M.M. (2020). The prevalence and risk factors of diabetic retinopathy in selected primary care centers during the 3-year screening intervals. Journal of Family Medicine and Primary Care, 7(5): 975-981. https://doi.org/10.4103/jfmpc.jfmpc_85_18

[5] Figueiredo, I.N., Kumar, S., Oliveira, C.M., Oliveira, C.M., Engquist, B. (2015). Automated lesion detectors in retinal fundus images. Computers in Biology and Medicine, 66: 47-65. https://doi.org/10.1016/j.compbiomed.2015.08.008

[6] Ministry of Health Diabetic Retinopathy Screening Team. (2017). Diabetic Retinopathy Screening Module 5 Handbook: Guide to Diabetic Retinopathy Screening MOH Diabetic Retinopathy.

[7] Taylor, R., Batey, D. (2012). Handbook of Retinal Screening in Diabetes. John Wiley & Sons, Ltd, 1-176. https://doi.org/10.1002/0470028858 

[8]  Rahim, S.S., Palade, V., Shuttleworth, J., Jayne, C. (2016). Automatic screening and classification of diabetic retinopathy and maculopathy using fuzzy image processing. Brain Informatics, 3(4): 249-267. https://doi.org/10.1007/s40708-016-0045-3

[9] Kamble, V.V., Kokate, R.D. (2020). Automated diabetic retinopathy detection using radial basis function. Procedia Computer Science, 167: 799-808. https://doi.org/10.1016/j.procs.2020.03.429

[10] Bellemo, V., Lim, Z.W., Lim, G., Nguyen, Q.D., Xie, Y., Yip, M.Y.T., Hamzah, H., Ho, J., Lee, X.Q., Hsu, W., Lee, M.L., Musonda, L., Chandran, M., Chipalo-Mutati, G., Muma, M., Tan, G.S.W., Sivaprasad, S., Menon, G., Wong, T.Y., Ting, D.S.W. (2019). Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: A clinical validation study. The Lancet Digital Health, 1(1): e35-e44. https://doi.org/10.1016/S2589-7500(19)30004-4

[11] Gargeya, R., Leng, T. (2017). Automated identification of diabetic retinopathy using deep learning. Ophthalmology, 124(7): 962-969. https://doi.org/10.1016/j.ophtha.2017.02.008

[12] Shankar, K., Sait, A.R.W., Gupta, D., Lakshmanaprabu, S.K., Khanna, A., Pandey, H.M. (2020). Automated detection and classification of fundus diabetic retinopathy images using synergic deep learning model. Pattern Recognition Letters, 133: 210-216. https://doi.org/10.1016/j.patrec.2020.02.026

[13] Xie, Y., Nguyen, Q.D., Hamzah, H., Lim, G., Bellemo, V., Gunasekeran, D.V., Yip, M.Y.T., Qi Lee, X., Hsu, W., Li Lee, M., Tan, C.S., Tym Wong, H., Lamoureux, E.L., Tan, G.S.W., Wong, T.Y., Finkelstein, E.A., Ting, D.S.W. (2020). Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: An economic analysis modelling study. The Lancet Digital Health, 2(5): e240-e249. https://doi.org/10.1016/S2589-7500(20)30060-1

[14] Kondakov, A., Kulik, S. (2020). Intelligent information system for telemedicine. Procedia Computer Science, 169: 240-243. https://doi.org/10.1016/j.procs.2020.02.142

[15] Chanchal, R., Chaman, J., Wasim, A. (2017). HEMAN: Health monitoring and nous: An IoT based e-health care system for remote telemedicine. 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Chennai, India. https://doi.org/10.1109/WiSPNET.2017.8300134

[16] Rezaeibagha, F., Mu, Y. (2018). Practical and secure telemedicine systems for user mobility. Journal of Biomedical Informatics, 78: 24-32. https://doi.org/10.1016/j.jbi.2017.12.011

[17] Pronsawatchai, P., Auefuea, S., Nartthanarung, A., Soontornpipit, P. (2018). Design of the electronic consultation system: Rama health electronic consulting. IEECON 2018 - 6th International Electrical Engineering Congress, Krabi, Thailand, pp. 1-4. https://doi.org/10.1109/IEECON.2018.8712302

[18] Liddy, C., Drosinis, P., Keely, E. (2016). Electronic consultation systems: Worldwide prevalence and their impact on patient care-a systematic review. Family Practice, 33(3): 274-285. https://doi.org/10.1093/fampra/cmw024

[19] Boegl, K., Adlassnig, K.P., Hayashi, Y., Rothenfluh, T.E., Leitich, H. (2004). Knowledge acquisition in the fuzzy knowledge representation framework of a medical consultation system. Artificial Intelligence in Medicine, 30(1): 1-26. https://doi.org/10.1016/S0933-3657(02)00073-8

[20] Banks, J., Farr, M., Edwards, H., Horwood, J., Salisbury, C., Northstone, K., Bernard, E. (2018). Use of an electronic consultation system in primary care: A qualitative interview study. British Journal of General Practice, 68(666): e1-e8. https://doi.org/10.3399/bjgp17X693509

[21] Black, S.W. (2015). Current Practices for Product Usability Testing in Web and Mobile Applications. 51. http://scholars.unh.edu/cgi/viewcontent.cgi?article=1228&context=honors.

[22] Christian Osterbauer, Monika Köhle, Thomas Grechenig. (2000). Web Usability Testing - A case study of usability testing of chosen sites (banks, daily newspapers, insurances). Conference: AusWeb2K, Sixth Australian World Wide Web Conference. http://ausweb.scu.edu.au/aw2k/papers/osterbauer/paper.html.

[23] Sneed, H. (2000). Testing software for internet applications. Wiley Online Library, 1(1): 15-22. https://doi.org/10.1002/1529-7950(200009)1:1<15::AID-SWF3>3.0.CO;2-8

[24] Sneed, H.M. (2004). Testing a web application. Proceedings - Sixth IEEE International Workshop on Web Site Evolution, Chicago, IL, USA, pp. 3-10. https://doi.org/10.1109/WSE.2004.10011

[25] Expero Staff. (2012). Usefulness Vs. Usability. Expero. https://www.experoinc.com/post/usefulness-vs-usability.

[26] Ambalov, I.A. (2021). Decomposition of perceived usefulness: A theoretical perspective and empirical test. Technology in Society, 64: 101520. https://doi.org/10.1016/j.techsoc.2020.101520

[27] McCloskey, D.W. (2006). the importance of ease of use, usefulness, and trust to online consumers. Journal of Organizational and End User Computing, 18(3): 47-65. https://doi.org/10.4018/joeuc.2006070103

[28] Lauesen, S., Younessi, H. (1988). Six styles for usability requirements. Proceedings of REFSQ’98, Presses Universitaires de Namur, pp. 1-12. https://www.itu.dk/~slauesen/Papers/SixStyles.pdf.

[29] McKenzie, K., Edgell, D. (2009). Role reversal: testing for ease of learning and recall. User Experience Magazine. https://uxpamagazine.org/testing_ease_learning_recall/.

[30] Mifsud, J. (n.d.). Usability Metrics – A Guide to Quantify the Usability of Any System. Usability Geek. https://usabilitygeek.com/usability-metrics-a-guide-to-quantify-system-usability/, accessed on June 9, 2021.

[31] Rinder, M., Writer, S., Friserv. (2012). The importance of website usability testing. Applied Information Management, 1277(800): 1-141.

[32] Ministry of Health Malaysia. (2017). Diabetic Retinopathy Screening Module KKM. https://www.moh.gov.my/moh/resources/Penerbitan/2017/Rujukan/NCD%202017/Diabetic_Retinopathy_Screening_Module.pdf.

[33] A’bas, N.N., Rahim, S.S., Dolhalit, M.L., Saifudin, W.S.N., Abdullasim, N., Parumo, S., Raja Omar, R.N., Md Khair, S.Z., Kalaichelvam, K., Noor Izhar, S.I. (2021). Development and usability testing of a consultation system for diabetic retinopathy screening. International Journal of Advanced Computer Science and Applications, 12(5): 178-188. https://doi.org/10.14569/IJACSA.2021.012052