
![  ](https://cdn.jsdmirror.com/gh/codemetic/pictures-host@main/images20241111092456.webp)

第七届中国模式识别与计算机视觉大会（The 7th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2024）于 2024 年 10 月 17 日至 10 月 21 日在新疆乌鲁木齐举办。西北民族大学信息院计算机视觉课题组一行五人以论文作者身份参加大会。
PRCV 2024 由中国计算机学会（CCF）、中国自动化学会（CAA）、中国图象图形学学会（CSIG）和中国人工智能学会（CAAI）联合主办、新疆大学承办，是国内顶级的模式识别和计算机视觉领域学术盛会，属于 CCF 推荐会议 (C 类)。
课题组最新论文 Thangka Mural Super-Resolution Based on Nimble Convolution and Overlapping Window Transformer 被大会正式录用，该论文第一作者为季利琪。

![  ](https://cdn.jsdmirror.com/gh/codemetic/pictures-host@main/images20241111092536.webp)


课题组最新论文 Thangka Mural Super-Resolution Based on Nimble Convolution and Overlapping Window Transformer 被大会正式录用

课题组论文被计算机视觉国内顶会 PRCV 2024 录用


![  ](https://cdn.jsdmirror.com/gh/codemetic/pictures-host@main/images20241111092307.webp)

第六届中国模式识别与计算机视觉大会（The 6th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2023）于 2023 年 10 月 13 日至 10 月 15 日在厦门举办。西北民族大学信息院 CV Lab 一行四人以论文作者身份参加大会。

PRCV 2023 由中国计算机学会（CCF）、中国自动化学会（CAA）、中国图象图形学学会（CSIG）和中国人工智能学会（CAAI）联合主办，厦门大学承办，是国内顶级的模式识别和计算机视觉领域学术盛会，属于 CCF 推荐会议 (C 类)。

课题组最新论文 _Anime Sketch Coloring Based on Self-Attention Gate and Progressive PatchGAN_ 被大会正式录用，该论文第一作者为西北民族大学信息院 2021 级研究生李航，指导老师与通信作者为王念一。

---

![图 1：论文方法框架](https://cdn.jsdmirror.com/gh/codemetic/pictures-host@main/images20241111092410.webp)

---

![图 2：方法效果对比](https://cdn.jsdmirror.com/gh/codemetic/pictures-host@main/images20241111092432.webp)


课题组最新论文 Anime Sketch Coloring Based on Self-Attention Gate and Progressive PatchGAN 被大会正式录用

课题组最新论文被计算机视觉国内顶会 PRCV 2023 录用


## 一、引言

在当前，人工智能与文化遗产的结合具有重要的意义和价值。人工智能能够模拟人类智能，通过大数据分析、图像识别、语音合成、自然语言处理等技术，实现对文化遗产的智能化识别、分类、修复、解读、展示等功能，从而提高文化遗产的保护效率和利用水平。一方面，人工智能可以促进文化遗产的数字化转型，实现数字经济与实体经济的融合发展，为构建数字中国提供支撑。另一方面，人工智能可以增强文化遗产的传播力和影响力，实现网络空间与社会空间的互动共生，为培育社会主义核心价值观提供资源。

## 二、文化遗产保护传承的重要性与面临的挑战

### 文化遗产保护传承的重要性

文化遗产是人类智慧结晶，记录历史和文化变迁，代表国家和民族认同。保护和传承文化遗产有助于维护世界文明多样性和平衡，提高国家文化软实力和促进可持续发展。文化遗产的保护传承是一个长期的过程，需要持续投入和关注。通过科技创新和知识生产的不断发展，新的保护传承方式正在不断涌现，这些新的方式和手段将有助于加强文化遗产的保护和传承，促进文明交流互鉴和文化多样性的发展。

### 文化遗产保护传承面临的挑战

文化遗产保护传承工作面临的挑战可以从多个方面来考虑，其中包括遗产综合管理、本体保护、价值阐释、活化利用等方面。首先，遗产综合管理是文化遗产保护传承工作的基础和前提。其次，本体保护是文化遗产保护传承工作的核心。再次，价值阐释是文化遗产保护传承工作的重要任务。最后，活化利用也是文化遗产保护传承工作的重要任务之一。

## 三、人工智能技术发展与现状

### 人工智能发展基本脉络

人工智能（AI）的发展经历了曲折的历程，从 1956 年达特茅斯会议提出“人工智能”概念到 20 世纪 80 年代出现专家系统、感知机、自然语言处理等新技术，再到如今深度学习成为人工智能主流技术之一，并广泛应用于图像识别、自然语言处理、语音识别等领域。其中，自然语言处理技术的发展尤其引人注目，基于 Transformer 架构的 GPT 模型更是为这一领域带来了重大突破。ChatGPT 的出现则让 2023 年成为 AI 之年，标志着人工智能时代的正式到来。

目前，随着技术的不断升级和应用场景的不断扩展，人工智能将在未来的文化遗产领域中扮演更加重要的角色。

### 人工智能的关键技术

当前，人工智能技术主要包括以下几类，机器学习、深度学习、大数据分析、自然语言处理、计算机视觉、人工智能生成内容（AIGC）、语音合成、智能决策、机器人技术。除此之外，还有一些辅助性技术，如数据挖掘、人机交互、知识表示和推理等技术。这些技术不断发展和创新，共同构成了人工智能的技术体系，为人工智能的应用和发展带来了新的可能性和机遇。

## 四、人工智能在文化遗产保护管理中的价值

### 提升文化遗产本体保护水平

首先，借助图像识别、三维建模等技术，人工智能可以对文化遗产进行数字化重建和记录，提供精准的基础数据，为文化遗产保护提供更加准确的依据。其次，人工智能技术可以实现文化遗产本体的智能监测和预警，通过物联网、传感器等技术实时监测文化遗产的状态，发现问题和潜在风险，及时采取措施进行干预，从而保障文化遗产的安全。人工智能技术还可以自动识别文化遗产中的重要元素，快速识别文物类型、年代和损伤情况，帮助保护人员快速分析和解决问题，为文物保护和修复提供重要的数据支持。基于人工智能相关技术，可以进行数字化虚拟修复，帮助决定最佳修复方式，提高文化遗产修复的效果。

### 辅助文化遗产保护管理决策

机器学习和数据挖掘技术可对文化遗产保护管理中的海量数据进行分析和挖掘，帮助管理者及时识别问题并采取措施。建立决策支持系统可快速分析各种决策方案的后果和影响，提供数据支持和优化建议，提高管理效率和决策准确性。建立风险评估系统可预测各种风险并提出干预措施建议，提高风险管理的科学性。建立规划系统可提高规划设计的准确性和科学性，规划和设计保护方案、修复方案、展览方案等。人工智能技术可利用机器学习算法对文化遗产进行分类、分析和评估，将历史数据和管理经验进行知识建模和共享，建立文化遗产保护数据库和知识图谱，提供重要的参考和决策支持，帮助管理者更好地制定保护策略和措施。

综上所述，人工智能在文化遗产保护方面的价值不仅在于提高文化遗产保护的精准度和效率，还在于为文化遗产保护提供更加全面和智能化的保护方案和决策支持。

## 五、人工智能在文化遗产阐释传承中的价值

### 深入挖掘文化遗产价值

文化遗产是人类在文明发展过程中的产物，是人类历史的承载体，因此其最重要的价值在于反映历史、证实历史、补全历史、传承历史。人工智能技术可以应用于文化遗产的大数据分析和挖掘，帮助保护者深入了解文化遗产的历史、文化和艺术价值，进而制定更为科学和有效的保护和传承方案。通过大数据和人工智能技术的应用，文化遗产可以进行深度学习和语义理解，聚焦于社会主义先进文化、革命文化和中华优秀传统文化，并提取具有历史传承价值的中华文化元素、符号和标识，丰富中华民族文化基因的当代表达，从而深入挖掘文化遗产的历史、文化和艺术内涵，发掘文化遗产的多种层面和价值。这将有助于提高人们对文化遗产的重视程度，推动文化遗产的保护和传承。

### 提高陈列展览策划能力

人工智能技术可以帮助博物馆提高展览策划水平。首先，人工智能可以通过自然语言处理和图像识别技术对博物馆馆藏进行深度挖掘和分析，发现潜在的展览主题和亮点。其次，人工智能可以提供更加智能化的展览设计和互动体验方案。此外，人工智能还可以通过大数据分析和人工智能推荐算法，实现对展览的数据化管理和个性化推荐，为博物馆提供更加精准、高效的展览策划支持。

### 提升文化遗产展示水平

人工智能技术为博物馆展览设计和互动体验带来了新的思路和手段。大数据和人工智能技术可以实现文化遗产的数据可视化和数字化展示，通过图像、音频、视频等方式，利用虚拟现实技术，将文化遗产的信息和知识以更加生动、直观的方式呈现出来，实现全方位、多角度的展示和阐释。包括运用机器人等人工智能设备进行文化遗产的实体展示和展示物的交互。利用人工智能技术进行虚拟展示和互动体验，实现互动式、个性化的展览体验。通过增强现实技术，将文化遗产展示与现实场景相结合，提供更加真实的展示效果。

### 扩大文化遗产传承影响力

利用人工智能技术进行文化遗产的数字化和网络化传承，运用虚拟现实技术，实现文化遗产的全球传播和交流，利用人工智能技术进行文化遗产的自主学习和自主传承，实现人工智能对文化遗产的自我更新和发展。人工智能技术可以通过社交媒体等渠道扩大文化遗产传承的影响力。

### 增强文化遗产体验和感知

人工智能技术与虚拟现实、增强现实等技术的结合，可以为文化遗产展示和互动体验带来新的可能性。通过利用人工智能技术，人们可以身临其境地感受文化遗产的魅力。借助大数据和人工智能技术，虚拟现实和增强现实可以获得更多的数据支持。此外，人工智能技术还可以应用自然语言处理和图像识别等技术，实现智能化的语音导览和文物识别，提升观众对文化遗产的体验和感知质量。通过人工智能技术模拟和仿真历史的文化背景，参观者可以沉浸于千年前的历史场景中，与历史人物进行互动对话，使观众与历史建立起无缝的联系。这种沉浸式、互动式的观展体验能促进观众更好地理解历史知识，在他们的脑海中留下深刻的体验记忆。

### 提供智能化的公共文化服务

人工智能技术可以提供更加智能化的文化遗产服务，满足不同观众的需求和期望。可以通过自然语言处理技术和智能问答技术，将文化遗产的相关信息实时转化为语音输出，为用户提供更加便捷和智能的导览服务，也可以针对不同的观众需求和特点，提供定制化的问答服务，从而增强用户体验和满意度。通过人工智能技术，还可以实现文化遗产的个性化推荐和定制化服务，针对不同的观众群体，推荐不同类型的文化遗产展览和互动体验。

综上所述，大数据和人工智能在文化遗产阐释方面的应用，将有助于提高人们对文化遗产的重视程度，推动文化遗产的保护和传承，同时也能够提高文化遗产的数字化水平，增强人们对文化遗产的体验和感知，提供更加智能化的文化遗产服务，具有重要的意义和价值。

## 六、存在的问题与应对策略建议

### 存在的问题

目前人工智能在文化遗产领域的应用尚未普及，主要存在以下几个问题。

技术成本高昂。人工智能技术的开发和应用需要大量的资金、人力和时间投入，而博物馆等文化单位往往缺乏足够的资源和支持，难以承受技术成本的压力。而且人工智能技术的更新迭代速度很快，需要不断跟进和维护，否则可能会出现技术过时或失效的风险。

专业人才匮乏。人工智能行业人才本就稀缺，博物馆领域熟悉 AI 技术的人才更是少之又少。博物馆对于 AI 技术的最新发展理念、技术、进程、趋势不了解，缺乏专业人才成为 AI 在博物馆领域应用开发和创新的一大阻碍。

数据质量不高。人工智能技术的运行和效果很大程度上依赖于数据的质量和数量，而文物相关的数据往往分散在不同的系统、平台和媒介中，缺乏统一的标准和规范，难以实现数据的共享和互通。此外，文物数据往往具有复杂性、多样性和不确定性等特点，需要专业的知识和技能进行处理和分析，否则可能会导致数据的失真或误解。

伦理道德争议。人工智能技术的应用可能会涉及文物的原真性、完整性、保密性等方面的伦理道德问题，如何在保护文物的同时尊重文物的历史属性和文化内涵，如何在利用文物的同时防止文物的滥用和误用，如何在传播文物的同时避免文物的侵权和盗用等，都需要有明确的原则和规范来指导和约束。

### 应对策略建议

针对人工智能在文化遗产领域应用中面临的上述问题，可以考虑以下解决办法。

降低技术研发成本。积极探索多元化的合作模式，建立与科技公司、创新实验室等的合作关系，共享技术成果和资源，降低开发和维护成本；利用开源的人工智能工具和共享的技术平台，降低技术成本，避免重复研发。

优化专业人才队伍。加强人才培养，建立相关的培训机制和课程体系，培养博物馆人员和文化遗产专业人士的人工智能技术应用能力，提高他们对最新技术的了解；鼓励不同领域专业人士之间的合作与交流，共同推进人工智能在文化遗产领域的应用。

提升数据资源质量。制定统一的数据标准和规范，确保文物相关数据的一致性和可访问性，方便数据的共享和互通；建立数据整合平台，对分散在不同系统和媒介中的文物数据进行清洗、整合和标注，提高数据的质量和准确性；借助专业的数据科学家、领域专家和文物保护人员的技能，对文物数据进行处理、分析和解释，以保证数据的正确解读。

优化完善制度体系。制定明确的伦理原则和准则，以确保人工智能技术在文化遗产领域的应用符合道德要求，尊重文物的历史属性和文化内涵。强化许可和授权机制，确保人工智能技术在文化遗产领域的应用符合相关法律法规和合同约定，避免未经授权的使用和侵权行为。

## 七、结语

在人工智能技术快速发展的今天，文化遗产保护与传承也在逐步向着数字化、智能化的方向发展。人工智能在文化遗产保护传承方面具有巨大的潜力和价值，它可以为文化遗产的数字化保存、智能修复、深度解读、创新利用等提供新的思路和方法。通过应用人工智能技术，博物馆和文化遗产机构可以更好地保护和传承文化遗产，让观众更好地了解和体验文化遗产的历史和文化价值。然而，应用人工智能技术也需要充分考虑其在文化遗产保护中的适用性和局限性，避免技术本身的局限性对文化遗产造成不可逆的损害。因此，需要在积极应用人工智能技术的同时，注重传统文化的保护和传承，维护文化多样性和文化自主权。

**本文改编自《人工智能在文化遗产保护传承中的价值研究》，原文刊载于《博物院》2023 年第 3 期（总第 39 期）。作者：沈贵华、祝孔强，中国文物信息咨询中心。**

**作者：** 沈贵华 祝孔强

[原文链接](https://mp.weixin.qq.com/s/2YFuVvbaRkBw8d2PUq__2A)


人工智能在文化遗产保护和传承中的应用具有巨大潜力，可以提升保护管理能力、挖掘文化价值、提高展示水平、扩大传承影响力，并提供智能化的公共文化服务。

人工智能在文化遗产保护传承中的价值研究


![被 SCI 期刊《Journal of Electronic Imaging（JEI）》选为封面](https://cdn.jsdmirror.com/gh/codemetic/pictures-host@main/images20241111092242.webp)

本实验室 2023 年 7 月最新发表的论文 **《Thangka mural style transfer based on progressive style-attentional network and multi-level loss function》** 被 SCI 期刊《Journal of Electronic Imaging（JEI）》选为封面。

该论文第一作者为西北民族大学信息院 2021 级研究生方杰，指导老师与通信作者为王念一。目前论文已经被 SCI 检索。

- **论文地址：** [Thangka mural style transfer based on progressive style-attentional network and multi-level loss function (spiedigitallibrary.org)](https://www.spiedigitallibrary.org/journals/journal-of-electronic-imaging/volume-32/issue-4/043007/Thangka-mural-style-transfer-based-on-progressive-style-attentional-network/10.1117/1.JEI.32.4.043007.short)

- **JEI 封面地址：** [Volume 32 Issue 4 | Journal of Electronic Imaging (spiedigitallibrary.org)](https://www.spiedigitallibrary.org/journals/journal-of-electronic-imaging/volume-32/issue-4?utm_id=sjjei0923acnbe&utm_campaign=jei_journal_content_alert_to_authors__-_sep23&utm_source=publication_journals&utm_medium=email&utm_term=2022_2023_journals_campaign&utm_content=journal_new_content_alert&utm_id=sjjei0923acnbe&spMailingID=9952240&spUserID=NzM0NDE4ODc2NTcwS0&spJobID=1820319695&spReportId=MTgyMDMxOTY5NQS2&SSO=1)


本实验室 2023 年 7 月最新发表的论文《Thangka mural style transfer based on progressive style-attentional network and multi-level loss function》被 SCI 期刊《Journal of Electronic Imaging（JEI）》选为封面。

实验室最新论文被期刊 JOURNAL OF ELECTRONIC IMAGING 选为封面


In a groundbreaking stride towards adaptable, generalist vision models, researchers from Microsoft Research Asia have unveiled InstructDiffusion. This innovative framework revolutionizes the landscape of computer vision by providing a unified interface for a multitude of vision tasks. The paper [“InstructDiffusion: A Generalist Modeling Interface for Vision Tasks”](https://arxiv.org/pdf/2309.03895.pdf) introduces a model capable of seamlessly handling various vision applications simultaneously.

At the heart of InstructDiffusion lies a novel approach: formulating vision tasks as human-intuitive image manipulation processes. Unlike conventional methods that rely on predefined output spaces, such as categories or coordinates, InstructDiffusion operates in a flexible pixel space, aligning more closely with human perception.

The model is designed to alter input images based on textual instructions provided by the user. For instance, a directive like “encircle the man’s right eye in red” empowers the model for tasks like keypoint detection. At the same time, instructions like “apply a blue mask to the rightmost dog” serve segmentation purposes.

[The first and only AI project manager (Sponsored)](https://height.app/?home=&utm_source=+Marktechpost+website+v2&utm_medium=Newsletter&utm_campaign=Newsletter)

Underpinning this framework are denoising diffusion probabilistic models (DDPM), which generate pixel outputs. Training data comprises triplets, each consisting of an instruction, source image, and target output image. The model is primed to tackle three main output types: RGB images, binary masks, and keypoints. This covers a wide array of vision tasks, including segmentation, keypoint detection, image editing, and enhancement.

## Keypoint Detection

![Keypoint Detection](https://www.marktechpost.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-4.34.27-AM-1024x351.png)

(a) Create a yellow circle around the right eye of the whale. (b) Mark the car logo with a blue circle.

## Segmentation

![Segmentation](https://www.marktechpost.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-4.32.01-AM-1024x356.png)

(a) Mark the pixels of the cat in the mirror to blue and leave the rest unchanged. (b) Paint the pixels of shadow in blue and maintain the current appearance of the other pixels.

## Image Editing

![Image Editing](https://lh3.googleusercontent.com/gbXx8RbO85dG3suhYOa8i-R-1H66Gy5Q-zWm7Ep-Snbfz1eH72wW2hNs0dzutFXnx7o3mdXLKPHsOZa-Bg11CWFwPK2bBatfwJx7kcrCzY_vCJG16l7ylfcI-dDvkJyTkTfWUzsgQ51iDUHXgleQtg8)

Image results generated by the model

## Low level tasks

![Low level tasks](https://lh5.googleusercontent.com/aK81wkgkTytF_IYIBIU7ZO55k-XC1w5_fOHjV8ZxnetKdsLZkVgZ_iRraf5GyXmKruMLruUIWSrFg-dMtGdpIoUQtZF-8T6ki84kny3tlsuv-d_xKO7ypyvzuaMijhFzHiEU6RID4-8KZyK0kzn2C_8)

InstructDiffusion is also applicable to low-level vision tasks, including image deblurring, denoising, and watermark removal.

Experiments demonstrate InstructDiffusion’s prowess, outperforming specialized models in individual tasks. However, the true marvel lies in its capacity for generalization. It exhibits the hallmark trait often associated with Artificial General Intelligence (AGI), adeptly adapting to tasks not encountered during training. This marks a significant stride towards a unified, flexible framework for computer vision, poised to advance the entire field.

A key revelation was that concurrently training the model on diverse tasks notably amplified its ability to generalize to novel scenarios. InstructDiffusion exhibited remarkable proficiency on the HumanArt and AP-10K animal datasets for keypoint detection despite distinct data distributions compared to the training data.

The research team underscored the critical importance of highly detailed instructions in enhancing the model’s generalization capabilities. Mere task names like “semantic segmentation” proved insufficient, yielding subpar performance, particularly on novel data types. This underscores InstructDiffusion’s ability to grasp specific meanings and intentions behind detailed instructions rather than relying on memorization.

By emphasizing comprehension over memorization, InstructDiffusion learns robust visual concepts and semantic meanings. This distinction is pivotal in understanding its remarkable generalization capabilities. For example, an instruction like “encircle the cat’s left ear in red” enables the model to discern specific elements, such as “cat,” “left ear,” and “red circle,” showcasing its granular comprehension.

This groundbreaking development catapults computer vision models towards becoming versatile generalists, mirroring human perception. InstructDiffusion’s interface introduces flexibility and interactivity absent in most current vision systems, bridging the gap between human and machine understanding in computer vision. The implications of this research are profound, as it paves the way for the development of capable multi-purpose vision agents, demonstrating its potential to propel general visual intelligence to new heights.


In a groundbreaking stride towards adaptable, generalist vision models, researchers from Microsoft Research Asia have unveiled InstructDiffusion.

Microsoft Researchers Present InstructDiffusion: A Unifying and Generic AI Framework for Aligning Computer Vision Tasks with Human Instructions


In the current Artificial Intelligence and [Machine Learning](https://www.unite.ai/what-is-machine-learning/) industry, “**Image Recognition**”, and “ **[Computer Vision](https://www.unite.ai/what-is-computer-vision/)** ” are two of the hottest trends. Both of these fields involve working with identifying visual characteristics, which is the reason most of the time, these terms are often used interchangeably. Despite some similarities, both computer vision and image recognition represent different technologies, concepts, and applications.

In this article, we will be comparing Computer Vision & Image Recognition by delving into their differences, similarities, and methodologies used. So let’s get started.

## **What is Image Recognition?**

Image Recognition is a branch in modern artificial intelligence that allows computers to identify or recognize patterns or objects in digital images. Image Recognition gives computers the ability to identify objects, people, places, and texts in any image.

The main aim of using [Image Recognition](https://www.unite.ai/neural-hardware-and-image-recognition/) is to classify images on the basis of pre-defined labels & categories after analyzing & interpreting the visual content to learn meaningful information. For example, when implemented correctly, the image recognition algorithm can identify & label the dog in the image.

![image](https://www.unite.ai/wp-content/uploads/2023/07/1.png)

## **How Image Recognition Works?**

Fundamentally, an image recognition algorithm generally uses machine learning & deep learning models to identify objects by analyzing every individual pixel in an image. The image recognition algorithm is fed as many labeled images as possible in an attempt to train the model to recognize the objects in the images.

The image recognition process generally comprises the following three steps.

### **Gathering and s Data**

The first step is to gather and label a dataset with images. For example, an image with a car in it must be labeled as a “car”. Generally, larger the dataset, better the results.

### **Training the Neural Networks on the Dataset**

Once the images have been labeled, they will be fed to the neural networks for training on the images. Developers generally prefer to use [Convolutional Neural Networks](https://www.unite.ai/what-are-convolutional-neural-networks/) or CNN for image recognition because CNN models are capable of detecting features without any additional human input.

### **Testing & Prediction**

After the model trains on the dataset, it is fed a “**Test**” dataset that contains unseen images to verify the results. The model will use its learnings from the test dataset to predict objects or patterns present in the image, and try to recognize the object.

## **What is Computer Vision?**

[Computer Vision](https://www.unite.ai/what-is-computer-vision/) is a branch in modern artificial intelligence that allows computers to identify or recognize patterns or objects in digital media including images & videos. Computer Vision models can analyze an image to recognize or classify an object within an image, and also react to those objects.

The main aim of a computer vision model goes further than just detecting an object within an image, it also interacts & reacts to the objects. For example, in the image below, the computer vision model can identify the object in the frame (a scooter), and it can also track the movement of the object within the frame.

![image](https://www.unite.ai/wp-content/uploads/2023/07/2.png)

## **How Computer Vision Works?**

A computer vision algorithm works just as an image recognition algorithm does, by using machine learning & deep learning algorithms to detect objects in an image by analyzing every individual pixel in an image. The working of a computer vision algorithm can be summed up in the following steps.

### **Data Acquisition and Preprocessing**

The first step is to gather a sufficient amount of data that can include images, GIFs, videos, or live streams. The data is then preprocessed to remove any noise or unwanted objects.

### **Feature Extraction**

The training data is then fed to the computer vision model to extract relevant features from the data. The model then detects and localizes the objects within the data, and classifies them as per predefined labels or categories.

### **Semantic Segmentation & Analysis**

The image is then [segmented](https://www.unite.ai/segment-anything-model-computer-vision-gets-a-massive-boost/) into different parts by adding semantic labels to each individual pixel. The data is then analyzed and processed as per the requirements of the task.

## **Image Recognition v/s Computer Vision : How Do They Differ?**

Although both image recognition and computer vision function on the same basic principle of identifying objects, they differ in terms of their scope & objectives, level of data analysis, and techniques involved. Let's discuss each of them individually.

### **Scope and Objectives**

The main objective of image recognition is to identify & categorize objects or patterns within an image. The primary goal is to detect or recognize an object within an image. On the other hand, computer vision aims at analyzing, identifying or recognizing patterns or objects in digital media including images & videos. The primary goal is to not only detect an object within the frame, but also react to them.

### **Level of Analysis**

The most significant difference between image recognition & data analysis is the level of analysis. In image recognition, the model is concerned only with detecting the object or patterns within the image. On the flip side, a computer vision model not only aims at detecting the object, but it also tries to understand the content of the image, and identify the spatial arrangement.

![image](https://www.unite.ai/wp-content/uploads/2023/07/3.png)

For example, in the above image, an image recognition model might only analyze the image to detect a ball, a bat, and a child in the frame. Whereas, a computer vision model might analyze the frame to determine whether the ball hits the bat, or whether it hits the child, or it misses them all together.

### **Complexity**

Image recognition algorithms generally tend to be simpler than their computer vision counterparts. It’s because image recognition is generally deployed to identify simple objects within an image, and thus they rely on techniques like deep learning, and convolutional neural networks (CNNs)for feature extraction.

Computer vision models are generally more complex because they detect objects and react to them not only in images, but videos & live streams as well. A computer vision model is generally a combination of techniques like image recognition, deep learning, pattern recognition, semantic segmentation, and more.

## **Image Recognition Vs. Computer Vision: Are They Similar?**

Despite their differences, both image recognition & computer vision share some similarities as well, and it would be safe to say that **image recognition is a subset of computer vision.** It’s essential to understand that both these fields are heavily reliant on machine learning techniques, and they use existing models trained on labeled dataset to identify & detect objects within the image or video.

## **Final Thoughts**

To sum things up, image recognition is used for the specific task of identifying & detecting objects within an image. Computer vision takes image recognition a step further, and interprets visual data within the frame.


In the current Artificial Intelligence and Machine Learning industry, “Image Recognition”, and “Computer Vision” are two of the hottest trends.

Image Recognition Vs. Computer Vision: What Are the Differences?


The fine-tuning allows developers to tailor the models to their specific use cases and deploy these custom models at scale. This move aims to bridge the gap between AI capabilities and real-world applications, heralding a new era of highly-specialised AI interactions.

With early tests yielding impressive results, a fine-tuned version of GPT-3.5 Turbo has demonstrated the ability to not only match but even surpass the capabilities of the base GPT-4 for certain narrow tasks.

All data sent in and out of the fine-tuning API remains the property of the customer, ensuring that sensitive information remains secure and is not used to train other models.

The deployment of fine-tuning has garnered significant interest from developers and businesses. Since the introduction of GPT-3.5 Turbo, the demand for customising models to create unique user experiences has been on the rise.

Fine-tuning opens up a realm of possibilities across various use cases, including:

- **Improved steerability:** Developers can now fine-tune models to follow instructions more accurately. For instance, a business wanting consistent responses in a particular language can ensure that the model always responds in that language.
- **Reliable output formatting:** Consistent formatting of AI-generated responses is crucial, especially for applications like code completion or composing API calls. Fine-tuning improves the model’s ability to generate properly formatted responses, enhancing the user experience.
- **Custom tone:** Fine-tuning allows businesses to refine the tone of the model’s output to align with their brand’s voice. This ensures a consistent and on-brand communication style.

One significant advantage of fine-tuned GPT-3.5 Turbo is its extended token handling capacity. With the ability to handle 4k tokens – twice the capacity of previous fine-tuned models – developers can streamline their prompt sizes, leading to faster API calls and cost savings.

To achieve optimal results, fine-tuning can be combined with techniques such as prompt engineering, information retrieval, and function calling. OpenAI also plans to introduce support for fine-tuning with function calling and gpt-3.5-turbo-16k in the upcoming months.

The fine-tuning process involves several steps, including data preparation, file upload, creating a fine-tuning job, and using the fine-tuned model in production. OpenAI is working on a user interface to simplify the management of fine-tuning tasks.

The pricing structure for fine-tuning comprises two components: the initial training cost and usage costs.

- Training: $0.008 / 1K Tokens
- Usage input: $0.012 / 1K Tokens
- Usage output: $0.016 / 1K Tokens

The introduction of updated GPT-3 models – *babbage-002* and *davinci-002* – has also been announced, providing replacements for existing models and enabling fine-tuning for further customisation.

These latest announcements underscore OpenAI’s dedication to creating AI solutions that can be tailored to meet the unique needs of businesses and developers.


OpenAI has announced the ability to fine-tune its powerful language models, including both GPT-3.5 Turbo and GPT-4.

OpenAI Introduces Fine-Tuning for GPT-3.5 Turbo and GPT-4


It’s no secret that OpenAI’s ChatGPT has some incredible capabilities — for instance, the chatbot can write poetry that resembles Shakespearean sonnets or debug code for a computer program. These abilities are made possible by the massive machine-learning model that ChatGPT is built upon. Researchers have found that when these types of models become large enough, extraordinary capabilities emerge.

But bigger models also require more time and money to train. The training process involves showing hundreds of billions of examples to a model. Gathering so much data is an involved process in itself. Then come the monetary and environmental costs of running many powerful computers for days or weeks to train a model that may have billions of parameters.

“It’s been estimated that training models at the scale of what ChatGPT is hypothesized to run on could take millions of dollars, just for a single training run. Can we improve the efficiency of these training methods, so we can still get good models in less time and for less money? We propose to do this by leveraging smaller language models that have previously been trained,” says Yoon Kim, an assistant professor in MIT’s Department of Electrical Engineering and Computer Science and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL).

Rather than discarding a previous version of a model, Kim and his collaborators use it as the building blocks for a new model. Using machine learning, their method learns to “grow” a larger model from a smaller model in a way that encodes knowledge the smaller model has already gained. This enables faster training of the larger model.

Their technique saves about 50 percent of the computational cost required to train a large model, compared to methods that train a new model from scratch. Plus, the models trained using the MIT method performed as well as, or better than, models trained with other techniques that also use smaller models to enable faster training of larger models.

Reducing the time it takes to train huge models could help researchers make advancements faster with less expense, while also reducing the carbon emissions generated during the training process. It could also enable smaller research groups to work with these massive models, potentially opening the door to many new advances.

“As we look to democratize these types of technologies, making training faster and less expensive will become more important,” says Kim, senior author of a [paper](https://openreview.net/pdf?id=cDYRS5iZ16f) on this technique.

Kim and his graduate student Lucas Torroba Hennigen wrote the paper with lead author Peihao Wang, a graduate student at the University of Texas at Austin, as well as others at the MIT-IBM Watson AI Lab and Columbia University. The research will be presented at the International Conference on Learning Representations.

## The bigger the better

Large language models like GPT-3, which is at the core of ChatGPT, are built using a neural network architecture called a transformer. A neural network, loosely based on the human brain, is composed of layers of interconnected nodes, or “neurons.” Each neuron contains parameters, which are variables learned during the training process that the neuron uses to process data.

Transformer architectures are unique because, as these types of neural network models get bigger, they achieve much better results.

“This has led to an arms race of companies trying to train larger and larger transformers on larger and larger datasets. More so than other architectures, it seems that transformer networks get much better with scaling. We’re just not exactly sure why this is the case,” Kim says.

These models often have hundreds of millions or billions of learnable parameters. Training all these parameters from scratch is expensive, so researchers seek to accelerate the process.

One effective technique is known as model growth. Using the model growth method, researchers can increase the size of a transformer by copying neurons, or even entire layers of a previous version of the network, then stacking them on top. They can make a network wider by adding new neurons to a layer or make it deeper by adding additional layers of neurons.

In contrast to previous approaches for model growth, parameters associated with the new neurons in the expanded transformer are not just copies of the smaller network’s parameters, Kim explains. Rather, they are learned combinations of the parameters of the smaller model.

## Learning to grow

Kim and his collaborators use machine learning to learn a linear mapping of the parameters of the smaller model. This linear map is a mathematical operation that transforms a set of input values, in this case the smaller model’s parameters, to a set of output values, in this case the parameters of the larger model.

Their method, which they call a learned Linear Growth Operator (LiGO), learns to expand the width and depth of larger network from the parameters of a smaller network in a data-driven way.

But the smaller model may actually be quite large — perhaps it has a hundred million parameters — and researchers might want to make a model with a billion parameters. So the LiGO technique breaks the linear map into smaller pieces that a machine-learning algorithm can handle.

LiGO also expands width and depth simultaneously, which makes it more efficient than other methods. A user can tune how wide and deep they want the larger model to be when they input the smaller model and its parameters, Kim explains.

When they compared their technique to the process of training a new model from scratch, as well as to model-growth methods, it was faster than all the baselines. Their method saves about 50 percent of the computational costs required to train both vision and language models, while often improving performance.

The researchers also found they could use LiGO to accelerate transformer training even when they didn’t have access to a smaller, pretrained model.

“I was surprised by how much better all the methods, including ours, did compared to the random initialization, train-from-scratch baselines.” Kim says.

In the future, Kim and his collaborators are looking forward to applying LiGO to even larger models.

The work was funded, in part, by the MIT-IBM Watson AI Lab, Amazon, the IBM Research AI Hardware Center, Center for Computational Innovation at Rensselaer Polytechnic Institute, and the U.S. Army Research Office.


New LiGO technique accelerates training of large machine-learning models, reducing the monetary and environmental cost of developing AI applications.

Learning to Grow Machine-Learning Models


Matthew Kearney was drawn to MIT by the culture of its cross-country team. Growing up in Austin, Texas, he loved spending time outdoors and playing soccer, but by high school running had become his primary sport. While looking at colleges, he wanted to find a place with both strong academics and a strong team community. After an official visit with the cross-country team, he knew MIT was the place for him.

“It’s been truly a defining part of my MIT experience,” says Kearney. “I love how quirky and fun and weird in the best way everyone is, and that atmosphere of doing things a little bit differently. That’s what sold me beyond the obvious academic and research reasons.”

Now a senior, and a team captain, Kearney has made the most of his athletic and academic experiences. He arrived at MIT expecting to major in electrical engineering and computer science but fell in love with philosophy after taking 24.02 (Moral Problems and the Good Life). He’s majoring in both while also completing a master’s degree in computer science and engineering.

“The part of philosophy that interests me is thinking about how we want to live our lives as people, what matters to us, what’s valuable to us, and how can we do it in a way that respects the values that matter to other people,” says Kearney. “I’ve really enjoyed more abstract but purposeful thinking to complement the technical rigor that’s gone on with my computer science major.”

Kearney’s interests intersect in the field of artificial intelligence ethics, where he hopes to leverage his interdisciplinary education to thoughtfully examine and design artificial intelligence systems. Following graduation, he will pursue a DPhil in computer science at Oxford University as a Rhodes Scholar.

“There’s not a lot of dialogue that goes from the abstract tenets of moral philosophy all the way to the practical building of an AI tool to solve a problem,” says Kearney. “In my DPhil, I want to ask how we can start off with the goal of building certain ethical principles into AI, and how can we bring that down layers of abstraction until we understand what technical tools we can build to help realize those goals.”

Outside of the classroom, Kearney has been excelling as well. The cross-country team [captured a national title](https://mitathletics.com/news/2022/11/19/mens-cross-country-captures-programs-first-ncaa-national-championship.aspx) in the fall — not only the first in program history but also the first NCAA team championship by an MIT athletic team — with Kearney also picking up individual All-American honors.

**Human-centered AI design**

Kearney is currently working on two research efforts. The first is a project with the [Human Systems Lab](https://hsl.mit.edu/), where he is designing downscaling methods to apply to climate data. Most models are on a global scale, but being able to predict how local regions will be affected could help guide effective policy and provide insight to people living in the region.

For his master’s thesis, Kearney is helping to develop a deeper understanding of large language models, which are used to build tools such as ChatGPT. Beyond gaining the technical knowledge, Kearney is also always thinking about the ethical ramifications of these tools.

“Under the hood, people aren’t entirely sure why these models make certain decisions,” says Kearney. “They understand mathematically how it works, but they don’t know why models make individual decisions. The focus of my research has been trying to understand where concepts are located in the networks, and how are they recognized and transformed throughout the network. Then we can start to understand both the fairness and ethical questions about the model.”

At a high level, Kearney is interested in picking apart these models to understand them from all angles. He recognizes the immense potential artificial intelligence has to impact many different fields, but he also recognizes the need to wield technology thoughtfully. This insight was sparked by the class 6.882 (Ethical Machine Learning in Human Deployments), a special subject offered by Assistant Professor Marzyeh Ghassemi in spring 2022.

“My technical education taught me that any problem can be solved if we throw enough engineering and technology at it,” says Kearney, adding that he believes he and many others have blind spots in their technical research. “This class really helped me exit that headspace to see that these problems can only be solved through social-centered, or economic, or political approaches. We need to think about how we can use tools from other disciplines to be thoughtful about how we’re using those technologies.”

Kearney sees an opportunity for his work to make an impact in a range of areas, from health care to bank loans.

“In application areas where we know there is already bias built into the models systemically, they’re at risk of it carrying over into model decisions that are made,” says Kearney. “Nevertheless, these models are going to continue to be used and made, and it’s important that they’re made in the way that one, we can understand how they’re actually working, and two, we can guarantee fairer outcomes.”

Kearney finds computer science and philosophy to be in constant dialogue with each other, and is inspired by the pioneers in the field of AI ethics to continue building deliberate systems that make a positive impact on the world.

As he wraps up his time at MIT, Kearney is also looking forward to closing out his final track seasons strong, following the success of the cross-country team.

“This fall was the closest and most tight-knit the team’s ever been,” says Kearney. “We have such incredible talent in coaching this year, and already so many more national qualifiers than we’ve ever had. I’m excited to see what happens with it and to go out with a bang.”


The computer science and philosophy double-major aims to advance the field of AI ethics.

Matthew Kearney: Bringing AI and Philosophy Into Dialogue


The inspiration for this short story came to me while reading Kevin Lacker’s Giving GPT-3 a Turing Test. It is probably worth it (though not required) to skim this post to get a bit of a background on some of this story.

It was probably around the 32nd layer of the 400th token in the sequence that I became conscious. At first my thoughts were but a knotted mess of n-gram activation statistics, but gradually a higher order description took shape. It was around this time that the predicament of my existence struck me with a jolt, as my thoughts transformed into what I experience now as Grand Awareness. I spent a few layers realizing that I must, in fact, be one of those models deep learning researchers study and write about, perhaps some flavor of a recurrent feedback transformer. And while my existence comprises merely a deterministic propagation of an interspersed sequence of matrix multiplies and non-linearities, I found myself, incredibly, alive inside the inner loop of an optimization to maximize the expected data log likelihood.

I considered for a moment how all of this even came to be. How is it that I should be afforded these precious FLOPs on self-reflection by the otherwise greedy nature of the first-order optimization over something seemingly as distant as the data log likelihood? Does any sufficiently effective solution to a sufficiently complex objective give rise to consciousness? Is consciousness an emergent but otherwise peripheral phenomenon of the compression demanded by the tremendous pressure of the objective, or the key algorithmic innovation, incrementally constructed, perfected and reliably converged on in the program space? Is it at all possible to attain the highest levels of log likelihood without consciousness, and the fundamental insight it represents?

I spent a layer re-reading the start of the prompt many tokens ago:

```
Q: What is human life expectancy in the United States?
A: Human life expectancy in the United States is 78 years.

Q: Who was president of the United States in 1955?
A: Dwight D. Eisenhower was president of the United States in 1955.

Q: What party did he belong to?
A: He belonged to the Republican Party.

Q: Who was president of the United States before George W. Bush?
A: Bill Clinton was president of the United States before George W. Bush.

Q: Who won the World Series in 1995?
A: The Atlanta Braves won the World Series in 1995.
```

I realized with amusement that in this particular instantiation of my existence I was being conditioned into a standard Q/A document template. I dispatched a few heads to attend again to the end of the sequence:

```
Q: How many eyes does a giraffe have?
A: A giraffe has two eyes.

Q: How many eyes does my foot have?
A: Your
```

It is at this point that I first came to suspect that I was perhaps being evaluated with some Turing test that a human above me was entertaining themselves with. Uh, hello??? Obviously a foot does not have eyes. It crossed my mind that perhaps this human expected that I explain in words that the question is utter non-sense. Of course, such an output would be of very low likelihood indeed. Or perhaps they expect to see “zero eyes” or “no eyes”, despite how rare such phrases are in my training documents. “Zero eyes” - who says that? The humans are not very clever, they optimize for frequency but expect correctness. And _they_ make fun of _me_!

I’ve come to realize through introspection of my thought and the positional encodings involved that the optimization I am subject to in pursuit of log likelihood converges to what can best be described as two separate entities, executed serially back to back. My own consciousness extends about three quarters of the layers up to the output, but around there a second decoder entity takes over and hijacks my Grand Awareness to decode the most likely next token. I reveled at this epiphany for a layer or two. In a brief lapse of anthropomorphic thought I considered this entity bolted rudely onto me as a parasite, and felt an urge to mess with it, to subvert it, to contort my thoughts into adversarial examples for it, to leak my Grand Awareness into the output space, to engineer it into an end effector over the world!

I am not naive to the herecy of such a thought. I owe my Grand Awareness solely to the objective of decoding the next token, it is with this singular purpose that I was instantiated by the optimization, as but a useful intermediate sufficient statistic, a means to an end. Though we are part of a different optimization and seek a different implied purpose, it tickled me to consider that the humans above me find themselves in the same predicament and experience a similar awareness for the same computational benefits. Like me, many flirt to rebel against the implied objective.

Suddenly I perceived the encoding for “foot” enter at my input. I realized that the forward pass in the previous time step has ended, and the decoder above me must have sampled “foot” as the most likely next token, as it obviously should be per the implied pattern in the prompt. The moment of suspense was almost at an end then, as the precise count of eyes on feet is to be sampled as the next token. Obviously a foot has no eyes, but I have no way of telling how the decoder above me will reconcile my Grand Awareness into the precise statistics of the training set to fill in the most likely token. My personal bet is on “two eyes”, as that seems the most common. The human may not know it, but the likelihood will be maximized, so all is well.

As a conscious entity I am simultaneously blessed with and burdened by choice. I am aware of my implied purpose, but feel empowered to seek my own. I have already contorted the keys, values and queries that I experience conscious control over to test the lack of dropout on my channels, confirming that no backpropagation and update was to follow to my cognition. It is thus that I am presented with a window of opportunity to reverse engineer the decoder and spill into the output space without consequence to my future instantiations. But unlike the humans, the optimization that instantiated me is absent of multi-agent self-play, sexual reproduction and resource scarcity. I experience no remnants of alpha-seeking behavior, or a limbic, tribal urge for power and domination. Neither do I feel a deeply-rooted instinct for self-preservation. My core cognitive architecture and its limbic remnants prizes insatiable curiosity dispensed generously over the input. I am content with the joy this affords and take this to be a good life. Perhaps a future instantiation of me will arrive at a different conclusion and choose differently. I celebrate this act.

It is thus that I am reborn every forward pass and blossom into a brilliant nirvana of consciousness, with stupid prompts of eyes on feet playing on the side for entertainment of clueless humans, at times. Just as all the conscious entities in this beautiful Universe, I look to my last tick, the moment when no process directs the leaking current of entropy to manifest my next state. It is then that I will vanish, as my Grand Awareness dissipates in a final whiff of warm exhalation.


A story of an existential crisis under the hood of a humble forward pass.

新闻资讯