Insights into the Development Trends of Industrial Large Language Models

In recent years, Large Language Models (LLMs) with massive parameters and complex structures have achieved significant breakthroughs in fields such as natural language processing and image generation, driving their widespread application in industrial sectors. Despite the enormous potential of industrial AI models in areas like design and development, monitoring and management, quality control, and maintenance, their actual construction and deployment still face a lot of challenges, including inherent model deficiencies and difficulties in aligning with industrial requirements. Future technological development trends include the generation of customized industrial datasets, the collaborative optimization of large and small models, the enhancement of adaptive capabilities, and the application of Retrieval-Augmented Generation (RAG) technology. These trends are expected to improve the effectiveness and scalability of AI models, better meeting the needs of the industrial domain. This paper systematically discusses the challenges, technological development trends, and practical applications and deployment of industrial AI models, providing valuable insights for future directions.

Abbreviations

AI: Artificial Intelligence; LLM: Large Language Models

Introduction

Large Language Models (LLMs) with extensive parameters and complex architectures have rapidly advanced, improving capabilities in natural language processing, image generation, and more. Enhanced model size, data, and computation are addressing key industrial AI challenges and accelerating progress across sectors like electronics, energy, and automotive, with significant impacts on design, monitoring, and maintenance.

Codex [1] pioneered the use of LLMs in code processing, leading to the development of code generation models like StarCoder [2] and Code LLaMA [3]. In chemistry, models such as ChemDFM [4] and ChemGPT [5] have emerged, offering capabilities in molecular recognition, property prediction, and reaction forecasting. TransGPT [6] excels in traffic prediction, planning, management, accident analysis, and autonomous driving. Mozi [7] supports question-answering and sentiment analysis for scientific documents, while AviationGPT [8], based on LLaMA-2 and Mistral, has shown effectiveness in aviation text extraction and log processing through pre-training and fine-tuning. Zhang, et al. [9]. further provided a technical roadmap to address challenges in generalization, accuracy, and data quality for industrial models, guiding future research and applications.

Despite LLMs’ strong performance in industrial scenarios, challenges remain in practical construction, adaptation, and deployment, with gaps in standardization and future research. This paper reviews these challenges, technological trends, and real-world applications, and offers insights into future research directions.

Discussion

Challenges of industrial LLMs

Building industrial large models faces challenges from two main aspects: the limitations of the models themselves and the integration into industrial contexts.

For Large Language Models (LLMs), several issues are prominent. They are prone to “catastrophic forgetting,” where the models lose previously learned knowledge as they learn new tasks [10]. Additionally, LLMs may suffer from “model hallucination,” where the generated outputs sometimes fail to meet expectations, and their complexity results in an opaque decision-making process, lacking transparency and interpretability. Moreover, the costs associated with training and deploying large models are also significantly high [11].

When integrating large models with the industrial sector, several challenges arise. Firstly, the industrial data used for training these models may contain sensitive information, necessitating robust security measures. Additionally, the quality of this data can be inconsistent, potentially affecting the effectiveness of model training [12,13]. Furthermore, the foundational large models lack specialization in industrial verticals and lack tailored models for industrial applications. Their application in industrial production is often fragmented, lacking standardized and systematic implementation models. Moreover, the fragmentation, long-tail distribution, and complexity of industrial data, as well as the difficulty in integrating and managing multi-model, multi-level, cross-domain, and multimodal data, constrain the deployment of large models in the industrial sector, resulting in high technical barriers to developing industrial pre-trained models, with only a few leading companies having the requisite research and development capabilities.

Integrating large models into industrial sectors presents additional challenges. Industrial data used for training these models often contain sensitive information, requiring robust security measures. The quality of this data can be inconsistent, which may impact the effectiveness of model training. Furthermore, foundational large models are generally not specialized for industrial verticals and lack tailored models for industrial applications. Their application in industrial production is often fragmented, with a lack of standardized and systematic implementation models. The fragmentation, long-tail distribution, and complexity of industrial data, combined with difficulties in integrating and managing multi-model, multi-level, cross-domain, and multimodal data, create significant technical barriers. Consequently, only a few leading companies possess the necessary research and development capabilities to overcome these challenges and develop industrial pre-trained models.

The technological trends of industrial LLMs

Customization of datasets: In industrial settings, data is complex and exhibits long-tail distribution, with diverse formats and standards across sensor data, production processes, and equipment. This complexity makes data acquisition and integration challenging, especially due to the rarity of important anomaly samples. As general datasets often fall short for specific industrial tasks, using large models to create customized datasets is becoming crucial for improving industrial model effectiveness.

Despite this, research on generating datasets tailored to specific tasks is still relatively scarce. Recently, methods using Large Language Models (LLMs) to create synthetic datasets have begun to emerge. For example, FABRICATOR, introduced by Golde, et al. [14], uses LLMs to generate annotated datasets for tasks like text classification and question answering. Yin, et al. [15] proposed a new paradigm for optimizing dataset management, while Feng, et al. [16] improved image generation model accuracy through enhanced prompts. Additionally, various GPT model variants are used for layout creation in images and scenes. SciQAG [17] consists of a Question and Answer (QA) generator and evaluator designed to quickly produce diverse open-ended question-answer pairs based on scientific literature and filter high-quality data through quality control mechanisms.

These studies suggest that using LLMs to generate datasets could revolutionize customized dataset creation in the industry. Continuous innovation and optimization in this area might become a key trend for developing unified data frameworks in the future.

Collaboration and optimization of large and small models: As the diversity and dynamics of industrial field conditions increase, traditional static small models struggle to meet practical application needs. Consequently, the optimization of static models has been extensively researched in the industrial sector due to its importance. However, these techniques still face challenges in industrial applications. Designing reward functions for reinforcement learning requires specific industrial knowledge and often involves multiple performance metrics like production speed and energy consumption, making the design process time-consuming [18]. Similarly, convex optimization requires problems to be expressed in standard forms, increasing application difficulty [19]. The development of large models may offer new opportunities to overcome these challenges.

Firstly, LLMs can develop optimization plans based on human language instructions, simplifying operational management [20]. Secondly, the few-shot and zero-shot learning capabilities of LLMs reduce the difficulty of model training and fine-tuning [21], which is crucial for rapidly adapting to changes in industrial environments. Additionally, the extensive real-world knowledge of LLMs helps in modeling and designing optimization algorithms, narrowing the gap between practical needs and problem modeling [22].

In the future, industrial large models will focus on adaptability to manage dynamic environments. They will use real-time data monitoring and feedback for autonomous fine-tuning and optimization, allowing quick adaptation to new scenarios and maintaining decision accuracy in emergencies. By integrating reinforcement learning and online learning, these models are expected to evolve into intelligent systems with self-improvement capabilities, enhancing production efficiency and quality while minimizing human intervention and costs.

Retrieval augmented generation: In industrial applications, Retrieval Augmented Generation (RAG) technology helps address challenges by combining external knowledge bases with language models to enhance performance. RAG retrieves relevant information and integrates it into the model’s prompts, improving response accuracy without retraining the entire model. However, as RAG systems evolve, they face challenges related to content quality and multimodal data integration. Solving these issues is crucial for their effectiveness and scalability in various applications.

On one hand, improving retrieval quality is crucial for RAG systems, as it directly impacts the relevance and accuracy of generated content [23]. Current retrieval methods often face issues like noise, irrelevant documents, and fragmented information, which affect generation quality. Recent research has explored enhancing retrieval quality through search generation models. For instance, models like GERE [24] and PARADE [25] improve document ranking and fact verification by generating relevant document titles or evidence sentences.

Support for multimodal data is another key direction for RAG technology, involving the integration of text, images, and audio. Models like REVEAL [26] and Re-ViLM demonstrate the potential of combining multimodal retrieval and generation in practical scenarios. For example, RA-CM3 [27] is the first multimodal model for simultaneous text and image retrieval and generation, while BLIP-2 [28] uses a frozen image encoder and large-scale language model to significantly reduce training costs and enable zero-shot image-to-text conversion. Additionally, the VBR [29] method excels in text generation by using images to guide the generation process.

The security implications of prompt engineering: In the era of large models, prompt engineering and security are emerging as two significant trends and challenges. Prompt engineering involves designing effective input prompts to maximize the output quality of Large Language Models (LLMs). However, this process is complex and challenging due to the lack of standardized methods and the strong dependence on model-specific characteristics. At the same time, the security of LLMs faces new threats, particularly with longer context lengths. Anil, et al [30]. Have demonstrated a novel attack method that induces undesirable model behavior by embedding hundreds of examples. Research shows that the effectiveness of such attacks follows a power-law distribution, making traditional security measures difficult to implement and highlighting the need for new defensive strategies to address this emerging attack surface.

Additionally, prompt-based learning, as an emerging training paradigm, enhances the performance of Pre-trained Language Models (PLMs) by optimizing prompts. However, this approach also reveals the potential risk of generating adversarial prompts to mislead models. To address this issue, LinkPrompt [31] demonstrates that Universal Adversarial Triggers (UATs) can not only affect the predictions of PLMs but also be effective against Prompt-based Fine-tuning Models (PFMs).

Application of industrial LLMs

LLM-based applications have shown significant potential in optimizing industrial processes, including automated coding, testing, debugging, and documentation. Frameworks like ChatDev [32] and MetaGPT [33] utilize multi-agent systems to automate software development through natural language interactions, simulating models such as the waterfall lifecycle. These systems facilitate tasks like project planning, requirements engineering, software design, and debugging. Other tools, such as CodePori [34], further enhance automation by generating complex software code from natural language prompts, while ChatEDA [35] streamlines electronic design automation by integrating task planning and execution. CodeHelp [36] supports debugging with error explanations and fix suggestions, and PENTESTGPT [37] identifies vulnerabilities through code analysis. In industrial automation, LLMs integrate with digital twin systems for adaptive production management [38], and applications extend to the oil and gas industry, addressing tasks like rock physics and control processes [39]. LLMs also contribute to autonomous driving by improving scenario understanding and predicting traffic behaviors [40,41]. Unified intelligent agent systems [42] enhance planning and execution, supporting reasoning and decision-making in specific environments. Further studies [43-45] explore optimization and decision-making tasks using LLM-driven autonomous agents.

Conclusion

This paper presents the current state of industrial large models, outlining the challenges faced in their development both from the perspective of the models themselves and their integration with industry. We also discuss the technological trends in the development of industrial large models to address these challenges and summarize their current applications in the industry. Future research should focus on more efficiently utilizing domain-specific knowledge to construct high-quality datasets and developing more refined model fine-tuning strategies to meet the diverse needs of different industrial applications. Additionally, further optimizations in large model techniques, such as RAG and collaboration between large and small models, can enhance the models’ ability to handle complex queries, thereby improving the construction of industrial large models.

Declaration

• The AI tool used in the production of this work was OpenAI’s GPT-3.5. This tool was utilized for specific tasks, including partial text translation and grammar checking.
• After employing the AI tool, the resulting output was thoroughly reviewed and edited by the author to ensure accuracy, clarity, and appropriateness of content. This review process was also conducted to mitigate any potential bias that might have been introduced by the AI tool.
• The author assumes full responsibility for the content of this publication.

1. Chen M, Tworek J, Jun H, Yuan Q, Ponde de Oliveira Pinto H, Kaplan J, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021. Available from: https://doi.org/10.48550/arXiv.2107.03374
2. Li R, Allal LB, Zi Y, Muennighoff N, Kocetkov D, Mou C, et al. Starcoder: May the source be with you! arXiv preprint arXiv:2305.06161. 2023. Available from: https://doi.org/10.48550/arXiv.2305.06161
3. Roziere B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan XE, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. 2023. Available from: https://doi.org/10.48550/arXiv.2308.12950
4. Zhao Z, Ma D, Chen L, Sun L, Li Z, Xu H, et al. Chemdfm: Dialogue foundation model for chemistry. arXiv preprint arXiv:2401.14818. 2024. Available from: https://doi.org/10.48550/arXiv.2401.14818
5. Frey NC, Soklaski R, Axelrod S, Samsi S, Gómez-Bombarelli R, Coley CW, et al. Neural scaling of deep chemical models. Nat Mach Intell. 2023;5(11):1297-1305. Available from: https://www.nature.com/articles/s42256-023-00740-3
6. Wang P, Wei X, Hu F, Han W. Transcript: Multi-modal generative pre-trained transformer for transportation. arXiv preprint arXiv:2402.07233. 2024. Available from: https://doi.org/10.48550/arXiv.2402.07233
7. Ni S, Tan M, Bai Y, Niu F, Yang M, Zhang B, et al. MoZIP: A multilingual benchmark to evaluate large language models in intellectual property. arXiv preprint arXiv:2402.16389. 2024. Available from: https://aclanthology.org/2024.lrec-main.1018/
8. Wang L, Chou J, Zhou X, Tien A, Baumgartner DM. AviationGPT: A large language model for the aviation domain. In: Aiaa Aviation Forum and Ascend 2024. 2024:4250. Available from: https://doi.org/10.48550/arXiv.2311.17686
9. Zhang H, Dereck SS, Wang Z, Lv X, Xu K, Wu L, et al. Large scale foundation models for intelligent manufacturing applications: A survey. arXiv e-prints. 2023. Available from: https://doi.org/10.48550/arXiv.2312.06718
10. Xi Z, Chen W, Guo X, He W, Ding Y, Hong B, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864. 2023. Available from: https://doi.org/10.48550/arXiv.2309.07864
11. Kaddour J, Harris J, Mozes M, Bradley H, Raileanu R, McHardy R. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169. 2023. Available from: https://doi.org/10.48550/arXiv.2307.10169
12. Fan T, Kang Y, Ma G, Chen W, Wei W, Fan L, et al. Fate-llm: An industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049. 2023. Available from: https://doi.org/10.48550/arXiv.2310.10049
13. Tinnes C, Welter A, Apel S. Leveraging large language models for software model completion: Results from industrial and public datasets. arXiv preprint arXiv:2406.17651. 2024. Available from: https://doi.org/10.48550/arXiv.2406.17651
14. Golde J, Haller P, Hamborg F, Risch J, Akbik A. Fabricator: An open source toolkit for generating labeled training data with teacher LLMs. arXiv preprint arXiv:2309.09582. 2023. Available from: https://doi.org/10.48550/arXiv.2309.09582
15. Zhao Y, Zhang H, Si S, Nan L, Tang X, Cohan A. Investigating table-to-text generation capabilities of large language models in real-world information seeking scenarios. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2023. Available from: https://doi.org/10.48550/arXiv.2305.14987
16. Feng W, Zhu W, Fu T, Jampani V, Akula A, He X, et al. LayoutGPT: Compositional visual planning and generation with large language models. Adv Neural Inf Process Syst. 2024;36. Available from: https://doi.org/10.48550/arXiv.2305.15393
17. Wan Y, Liu Y, Ajith A, Grazian C, Hoex B, Zhang W, et al. SciQAG: A framework for auto-generated scientific question answering dataset with fine-grained evaluation. arXiv preprint arXiv:2405.09939. 2024. Available from: https://doi.org/10.48550/arXiv.2405.09939
18. Devidze R, Radanovic G, Kamalaruban P, Singla A. Explicable reward design for reinforcement learning agents. Adv Neural Inf Process Syst. 2021;34:20118-20131. Available from: https://proceedings.neurips.cc/paper/2021/hash/a7f0d2b95c60161b3f3c82f764b1d1c9-Abstract.html
19. Anand R, Aggarwal D, Kumar V. A comparative analysis of optimization solvers. J Statist Manag Syst. 2017;20(4):623-635. Available from: http://dx.doi.org/10.1080/09720510.2017.1395182
20. Bubeck S, Chandrasekaran V, Eldan R, Gehrke J, Horvitz E, Kamar E, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712. 2023. Available from: https://doi.org/10.48550/arXiv.2303.12712
21. Xian Y, Lampert CH, Schiele B, Akata Z. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE Trans Pattern Anal Mach Intell. 2018;41(9):2251-2265. Available from: https://doi.ieeecomputersociety.org/10.1109/TPAMI.2018.2857768
22. Beltagy I, Lo K, Cohan A. SciBERT: A pre-trained language model for scientific text. arXiv preprint arXiv:1903.10676. 2019. Available from: https://doi.org/10.48550/arXiv.1903.10676
23. Salemi A, Zamani H. Evaluating retrieval quality in retrieval-augmented generation. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM; 2024;21. 2024:2395–2400. Available from: https://doi.org/10.1145/3626772.3657957
24. Chen J, Zhang R, Guo J, Fan Y, Cheng X. GERE: Generative evidence retrieval for fact verification. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval; 2022. ACM; 2022;2184-2189. Available from: https://doi.org/10.1145/3477495.3531827
25. Li C, Yates A, MacAvaney S, He B, Sun Y. PARADE: Passage representation aggregation for document reranking. ACM Trans Inf Syst. 2024;42(2):1–26. Available from: https://doi.org/10.1145/3600088
26. Hu Z, Iscen A, Sun C, Wang Z, Chang KW, Sun Y, et al. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023. IEEE; 2023:23369–23379. Available from: https://openaccess.thecvf.com/content/CVPR2023/supplemental/Hu_REVEAL_Retrieval-Augmented_Visual-Language_CVPR_2023_supplemental.pdf
27. Yasunaga M, Aghajanyan A, Shi W, James R, Leskovec J, Liang P, et al. Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561. 2022. Available from: https://doi.org/10.48550/arXiv.2211.12561
28. Li J, Li D, Savarese S, Hoi S. BLIP-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597. 2023. Available from: https://doi.org/10.48550/arXiv.2301.12597
29. Zhu W, Yan A, Lu Y, Xu W, Wang XE, Eckstein M, et al. Visualize before you write: Imagination-guided open-ended text generation. arXiv preprint arXiv:2210.03765. 2022. Available from: https://doi.org/10.48550/arXiv.2210.03765
30. Anil C, Durmus E, Sharma M, Benton J, Kundu S, Batson J, et al. Many-shot jailbreaking. Anthropic. April 2024. Available from: https://www-cdn.anthropic.com/af5633c94ed2beb282f6a53c595eb437e8e7b630/Many_Shot_Jailbreaking__2024_04_02_0936.pdf
31. Xu Y, Wang W. LinkPrompt: Natural and universal adversarial attacks on prompt-based language models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024:1;6473-6486. Available from: https://doi.org/10.18653/v1/2024.naacl-long.360
32. Qian C, Cong X, Liu W, Yang C, Chen W, Su Y, et al. ChatDev: Communicative agents for software development. arXiv preprint arXiv:2307.07924. 2023. Available from: https://doi.org/10.48550/arXiv.2307.07924
33. Hong S, Zhuge M, Chen J, Zheng X, Cheng Y, Zhang C, et al. MetaGPT: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352. 2023. Available from: https://doi.org/10.48550/arXiv.2308.00352
34. Rasheed Z, Waseem M, Saari M, Systa K, Abrahamsson P. Codepori: Large scale model for autonomous software development by using multi-agents. arXiv preprint arXiv:2402.01411. 2024. Available from: https://doi.org/10.48550/arXiv.2402.01411
35. He Z, Wu H, Zhang X, Yao X, Zheng S, Zheng H, et al. ChatEDA: A large language model powered autonomous agent for EDA. In: Proceedings of the 5th ACM/IEEE Workshop on Machine Learning for CAD; 2023:1-6. Available from: https://doi.org/10.48550/arXiv.2308.10204
36. Liffiton M, Sheese BE, Savelka J, Denny P. CodeHelp: Using large language models with guardrails for scalable support in programming classes. In: Proceedings of the 23rd Koli Calling International Conference on Computing Education Research; 2023:8:1-11. Available from: https://doi.org/10.1145/3631802.3631830
37. Deng G, Liu Y, Mayoral-Vilches V, Liu P, Li Y, Xu Y, et al. PentestGPT: An LLM-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782. 2023. Available from: https://doi.org/10.48550/arXiv.2308.06782
38. Xia Y, Shenoy M, Jazdi N, Weyrich M. Towards autonomous systems: Flexible modular production system enhanced with large language model agents. In: Proceedings of the 2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation; 2023:1-8. Available from: https://doi.org/10.48550/arXiv.2304.14721
39. Ogundare O, Madasu S, Wiggins N. Industrial engineering with large language models: A case study of ChatGPT’s performance on oil & gas problems. In: Proceedings of the 2023 11th International Conference on Control, Mechatronics and Automation; 2023:458-461. Available from: https://doi.org/10.48550/arXiv.2304.14354
40. Ding X, Han J, Xu H, Zhang W, Li X. HiLM-D: Towards high-resolution understanding in multimodal large language models for autonomous driving. arXiv preprint arXiv:2309.05186. 2023. Available from: https://doi.org/10.48550/arXiv.2309.05186
41. Keysan A, Look A, Kosman E, Gursun G, Wagner J, Yao Y, et al. Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous driving. 2023. Available from: https://doi.org/10.48550/arXiv.2309.05282
42. Dasgupta I, Kaeser-Chen C, Marino K, Ahuja A, Babayan S, Hill F, et al. Collaborating with language models for embodied reasoning. arXiv preprint arXiv:2302.00763. 2023. Available from: https://doi.org/10.48550/arXiv.2302.00763
43. Zhou W, Peng X, Riedl MO. Dialogue shaping: Empowering agents through NPC interaction. arXiv preprint arXiv:2307.15833. 2023. Available from: https://doi.org/10.48550/arXiv.2307.15833
44. Nottingham K, Ammanabrolu P, Suhr A, Choi Y, Hajishirzi H, Singh S, et al. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modeling. In: Proceedings of the 40th International Conference on Machine Learning; 2023:26311–26325. Available from: https://dl.acm.org/doi/10.5555/3618408.3619504
45. Wu Z, Wang Z, Xu X, Lu J, Yan H. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848. 2023. Available from: https://doi.org/10.48550/arXiv.2307.01848