Application of the Sheridan and Verplank automation level scale for an objective assessment of business intelligence tools

With the rapid growth in the variety of business intelligence tools, the choice between visual Low-Code platforms and command line interface tools working in combination with large language models is increasingly subjective, which increases the risk of errors when implementing solutions. The study proposes the use of the Sheridan and Verplank Levels of Automation scale as an objective metric for comparing business analytics tools, adapting I.G. Korablev’s operation analysis methodology for assessing the level of automation for manufacturing enterprises to the tasks of dashboard development and the use of software pipelines interacting with large language models. The materials and methods of the study have been presented on the basis of an experiment that examined the processes of developing dashboards and process diagrams on the Apache Superset and Draw.io platforms, respectively, as well as using a software pipeline based on JupyterLab, Dash, Plotly, and PlantUML tools with the use of large language models. Based on the experiment results, the results of using quantitative assessment of automation levels on the Sheridan and Verplank scales have been described and the ergodicity in terms of the “stability” of business intelligence tools outside of ideal conditions. The findings demonstrate that the use of the Sheridan and Verplank scale of automation levels can help eliminate subjectivity when choosing business intelligence tools and can become the basis for management decisions in the IT sphere.

1. Alamin M.A.A., Malakar S., Uddin G., Afroz S., Haider T.B., Iqbal A. An Empirical Study of Developer Discussions on Low-Code Software Development Challenges. arXiv. 2021. https://doi.org/10.1109/MSR52588.2021.00018

2. Braarud P.O. An efficient screening technique for acceptable mental workload based on the NASA Task Load Index-development and application to control room validation. International Journal of Industrial Ergonomics. 2020;76. DOI: https://doi.org/10.1016/j.ergon.2019.102904.

3. Braarud P.O. Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human-machine work. International Journal of Industrial Ergonomics. 2021;86. https://doi.org/10.1016/j.ergon.2021.103233

4. Fagbohun O., Harrison R.M., Dereventsov A. An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner’s Guide. arXiv. 2024.

5. Hernandez J.O., Fonseca i Casas P. Business Intelligence’s Self-Service Tools Evaluation. Technologies. 2022;4(10). https://doi.org/10.3390/technologies10040092

6. Korablev I.G. Assessment of the level of automation of business processes of the enterprise. Bulletin of Cherepovets State University. 2016;1:17–24.

7. Lopes J., Faria M., Santos M.F. Exploring trends and autonomy levels of adaptive business intelligence in healthcare: A systematic review. PLoS One. 2024;19. https://doi.org/10.1371/journal.pone.0302697

8. Moray N., Rodriguez D., Clegg B.A. Levels of Automation in Process Control. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2000;44:93–96. https://doi.org/10.1177/154193120004400125

9. Sahoo P., Singh A.K., Saha S., Jain V., Mondal S., Chadha A. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv. 2025.

10. Sheridan T.B., Verplank W.L., Brooks T.L. Human and Computer Control of Undersea Teleoperators. In: 14th Annual Conference on Manual Control. 1978. Pp. 343–357. https://doi.org/10.21236/ADA057655

11. Sheridan T.B. Humans and automation: system design and research issues. 2002.

12. Suarez D., Almeida F., Blanco V. Comprehensive analysis of energy efficiency and performance of ARM and RISC-V SoCs. The Journal of Supercomputing. 2024;80:12771–12789. https://doi.org/10.1007/s11227-024-05946-9

13. Syed S. Towards Autonomous Analytics: The Evolution of Self-Service BI Platforms with Machine Learning Integration. Journal of Artificial Intelligence and Big Data. 2022. https://doi.org/10.31586/jaibd.2022.1157