Is ChatGPT Better at Business Consulting than an Experienced Human Analyst? An Experimental Comparison of Solutions to a Strategic Business Problem

Dawid Jarco; Lukasz Sulkowski

doi:10.23762/FSO_VOL11_NO2_4

Published : 2023-06-30

Vol. 11 No. 2 (2023)

Is ChatGPT Better at Business Consulting than an Experienced Human Analyst? An Experimental Comparison of Solutions to a Strategic Business Problem

Dawid Jarco

https://orcid.org/0000-0002-1174-123X

Lukasz Sulkowski

https://orcid.org/0000-0002-1248-2743

DOI: https://doi.org/10.23762/FSO_VOL11_NO2_4

Abstract

Given the level of continued technological progress, AI language models have been the subject of enormous media attention. The popularity of such models skyrocketed in November 2022 with the introduction of the ChatGPT application. The impact of AI language models on areas such as research, healthcare and business is constantly being evaluated. However, an issue in which there is a clear research gap is the use of ChatGPT for business decision-making purposes. The aim of this article is, therefore, to investigate whether ChatGPT can be leveraged to make strategic business decisions. The methodology compares three paths towards a case for a strategic business decision by means of a checklist-based score: one conducted by a human alone; the second taken purely by ChatGPT with as little human interaction as possible; and third, the centaur model, where a human actively interacts with ChatGPT and asks critical questions. The findings were twofold. Firstly, ChatGPT is still far from being considered a quality business consultant in itself. Secondly, however, AI turned out to be extremely helpful in terms of creating a decision-making path. Hence, humans may leverage the information provided by ChatGPT to gain helpful insights for better business decision making.

Keywords:

ChatGPT, AI, decision making, LLM, strategic management, financial analysis

Details

References

Statistics

Authors

Download files

PDF

Citation rules

Jarco, D., & Sulkowski, L. (2023). Is ChatGPT Better at Business Consulting than an Experienced Human Analyst? An Experimental Comparison of Solutions to a Strategic Business Problem. Forum Scientiae Oeconomia, 11(2), 87–109. https://doi.org/10.23762/FSO_VOL11_NO2_4

Altmetric indicators

Cited by / Share

Licence

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

References

Apple Press Release (2022, March 8th). Apple unveils M1 Ultra, the world’s most powerful chip for a personal computer. Available at:

https://www.apple.com/newsroom/2022/03/apple-unveils-m1-ultra-the-worlds-most-powerful-chip-for-a-personal-computer/, access: 25.05.2023.

Astill, J., Dara, R. A., Fraser, E. D., Roberts, B., & Sharif, S. (2020). Smart poultry management: Smart sensors, big data, and the internet of things. Computers and Electronics in Agriculture, 170, 105291.

Bini, S. A. (2018). Artificial intelligence, machine learning, deep learning, and cognitive computing: what do these terms mean and how will they impact health care?. The Journal of arthroplasty, 33(8), 2358-2361.

Bordeleau, F. E., Mosconi, E., & de Santa-Eulalia, L. A. (2020). Business intelligence and analytics value creation in Industry 4.0: a multiple case study in manufacturing medium enterprises. Production Planning & Control, 31(2-3), 173-185.

Borji, A., & Itti, L. (2014). Human vs. computer in scene and object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 113-120).

Brants, T., Popat, A. C., Xu, P., Och, F. J., & Dean, J. (2007). Large language models in machine translation.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.

Bryman, A. (2016). Social research methods. Oxford University Press.

Burger, B., Kanbach, D. K., Kraus, S., Breier, M., & Corvello, V. (2023). On the use of AI-based tools like ChatGPT to support management research. European Journal of Innovation Management, 26(7), 233-241.

Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., ... & Raffel, C. (2021, August). Extracting Training Data from Large Language Models. In USENIX Security Symposium (Vol. 6).

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications.

Davey, A. (2022, December 8th). OpenAI Chatbot Spits Out Biased Musings, Despite Guardrails. Bloomberg. Available at: https://www.bloomberg.com/news/newsletters/2022-12-08/chatgpt-open-ai-s-chatbot-is-spitting-out-biased-sexist-results, access: 25.05.2023.

Doshi, R. H., Bajaj, S. S., & Krumholz, H. M. (2023). ChatGPT: temptations of progress. The American Journal of Bioethics, 1-3.

Duan, Y., Edwards, J. S., & Dwivedi, Y. K. (2019). Artificial intelligence for decision making in the era of Big Data–evolution, challenges and research agenda. International journal of information management, 48, 63-71.

Duncan, R. B. (1972). Characteristics of organizational environments and perceived environmental uncertainty. Administrative science quarterly, 313-327.

Eisenhardt, K. M., & Zbaracki, M. J. (1992). Strategic decision making. Strategic management journal, 13(S2), 17-37.

Elbanna, S. (2006). Strategic decision‐making: Process perspectives. International Journal of Management reviews, 8(1), 1-20.

Gorry, G. A., & Scott Morton, M. S. (1971). A framework for management information systems.

Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2018). When will AI exceed human performance? Evidence from AI experts. Journal of Artificial Intelligence Research, 62, 729-754.

Korzynski, P., Mazurek, G., Altmann, A., Ejdys, J., Kazlauskaite, R., Paliszkiewicz, J., Wach, K., & Ziemba, E. (2023a). Generative artificial intelligence as a new context for management theories: analysis of ChatGPT. Central European Management Journal, 31(1), https://doi.org/10.1108/CEMJ-02-2023-0091.

Korzynski, P., Kozminski, A.K., & Baczynska, A. (2023b). Navigating leadership challenges with technology: Un-covering the potential of ChatGPT, virtual reality, human capital management system, robotic process automation, and social media. International Entreprenership Review, 9(2), https://doi.org/10.15678/IER.2023.0902.xx

Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G. Z. (2019). XAI—Explainable artificial intelligence. Science robotics, 4(37), eaay7120.

Jauch, L. R., & Kraft, K. L. (1986). Strategic management of uncertainty. Academy of management review, 11(4), 777-790.

Lee, M., Liang, P., & Yang, Q. (2022, April). Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-19).

Lee, R. (2021). The effects of smart factory operational strategies and system management on the innovative performance of small-and medium-sized manufacturing firms. Sustainability, 13(6), 3087.

Mijwil, M. M., & Abttan, R. A. (2021). Artificial intelligence: a survey on evolution and future trends. Asian Journal of Applied Sciences, 9(2).

Minsky, M. (1961). Steps toward artificial intelligence. Proceedings of the IRE, 49(1), 8-30.

Niu, Y., Ying, L., Yang, J., Bao, M., & Sivaparthipan, C. B. (2021). Organizational business intelligence and decision making using big data analytics. Information Processing & Management, 58(6), 102725.

Paschen, U., Pitt, C., & Kietzmann, J. (2020). Artificial intelligence: Building blocks and an innovation typology. Business Horizons, 63(2), 147-155.

Pelissari, R., Oliveira, M. C., Abackerli, A. J., Ben‐Amor, S., & Assumpção, M. R. P. (2021). Techniques to model uncertain input data of multi‐criteria decision‐making problems: a literature review. International Transactions in Operational Research, 28(2), 523-559.

Saffman, M. (2016). Quantum computing with atomic qubits and Rydberg interactions: progress and challenges. Journal of Physics B: Atomic, Molecular and Optical Physics, 49(20), 202001.

Simon, H. A. (1987). Making management decisions: The role of intuition and emotion. Academy of Management Perspectives, 1(1), 57-64.

Sivarajah, U., Irani, Z., Gupta, S., & Mahroof, K. (2020). Role of big data and social media analytics for business to business sustainability: A participatory web context. Industrial Marketing Management, 86, 163-179.

Tung, R. L. (1979). Dimensions of organizational environments: An exploratory study of their impact on organization structure. Academy of management journal, 22(4), 672-693.

Tung, R. L., & Miller, E. L. (1990). Managing in the twenty-first century: The need for global orientation. MIR: Management International Review, 5-18.

Wach, K., Duong, C.D., Ejdys, J., Kazlauskaitė, R., Korzynski, P., Mazurek, G., Paliszkiewicz, J., & Ziemba, E. (2023). The dark side of generative artificial intelligence: A critical analysis of controversies and risks of ChatGPT. Entrepreneurial Business and Economics Review, 11(2), 7-24. https://doi.org/10.15678/EBER.2023.110201.

van Dis, E. A., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: five priorities for research. Nature, 614(7947), 224-226.

Vanian, J. (2022, December 13th). Why tech insiders are so excited about ChatGPT, a chatbot that answers questions and writes essays. CNBC. Available at: https://www.cnbc.com/2022/12/13/chatgpt-is-a-new-ai-chatbot-that-can-answer-questions-and-write-essays.html, access: 25.05.2023.

Varriale, V., Cammarano, A., Michelino, F., & Caputo, M. (2021). Sustainable supply chains with blockchain, IoT and RFID: A simulation on order management. Sustainability, 13(11), 6372.

Walker, W. E., Harremoës, P., Rotmans, J., Van Der Sluijs, J. P., Van Asselt, M. B., Janssen, P., & Krayer von Krauss, M. P. (2003). Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integrated assessment, 4(1), 5-17.

Wang, F. Y., Yang, J., Wang, X., Li, J., & Han, Q. L. (2023). Chat with ChatGPT on Industry 5.0: Learning and Decision-Making for Intelligent Industries. IEEE/CAA Journal of Automatica Sinica, 10(4), 831-834.

Wang, Y., Kim, J., & Kim, J. (2021). The financial impact of online customer reviews in the restaurant industry: A moderating effect of brand equity. International Journal of Hospitality Management, 95, 102895.

Wesendrup, K., & Hellingrath, B. (2023). Post-prognostics demand management, production, spare parts and maintenance planning for a single-machine system using Reinforcement Learning. Computers & Industrial Engineering, 109216.

Yin, R. K. (2014). Case study research: Design and methods. Sage Publications.

Dawid Jarco

dawid.jarco@wsb.edu.pl
https://orcid.org/0000-0002-1174-123X

Affiliation:

WSB University in Dabrowa Gornicza

Bio Statement (e.g., department and rank)

Dawid Jarco is an assistant professor at WSB University and a member of the Management Department. He specialises in digitalisation, business intelligence and smart technologies. His research interests also include strategic business decision making and empirical data applications. Furthermore, he works as an operational manager for Precisely, a global company specialising in data integrity. ORCID no. 0000-0002-1174-123X

Lukasz Sulkowski

lsulkowski@wsb.edu.pl
https://orcid.org/0000-0002-1248-2743

Affiliation:

WSB University in Dąbrowa Górnicza

Bio Statement (e.g., department and rank)

Łukasz Sułkowski is a professor of economic sciences and humanities, specialising in management in higher education. He heads the Department of Management of Higher Education Institutions at the Jagiellonian University and works as a Vice-Rector for Research and Personnel Development at WSB University in Dąbrowa Górnicza. He is also the President of the Management Board of PCG Polska. ORCID no. 0000-0002-1248-2743