CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

什么是CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

CLaMP 3 is a pioneering framework in music information retrieval (MIR), addressing the challenges of cross-modal and cross-lingual music search. By utilizing contrastive learning, CLaMP 3 can align various music representations—including sheet music, audio, and performance signals—across different languages and modalities. This allows users to retrieve music with high accuracy using textual descriptions in multiple languages. Through its innovative approach, CLaMP 3 enables music retrieval even when traditional methods fail due to the unavailability of aligned data.

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages怎么用？

To use CLaMP 3, users can input queries in various languages to retrieve corresponding music across different formats such as sheet music, audio, or performance signals. The framework employs a multilingual text encoder to process queries and match them to music representations. Additionally, users can also perform cross-modal music retrieval, such as finding audio or sheet music that corresponds to a specific image or video scene, enhancing the flexibility of music search.

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages核心功能

• CLaMP 3核心功能：
• 利用对比学习实现跨模态和跨语言的音乐检索
• 支持多语言文本编码器，适应未见语言的检索需求
• 基于M4-RAG数据集，提供超过230万对音乐-文本数据
• 支持零样本音乐分类，无需标签数据即可进行灵活分类
• 提供WikiMT-X基准数据集，推动音乐信息检索的进一步研究
• 图像到音乐的跨模态检索，通过视觉描述匹配音乐
• 通过t-SNE可视化展示CLaMP 3在学习的音乐表示上的分布

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages使用案例

• CLaMP 3使用案例：
• 使用文本查询如“经典摇滚，英国，1960年代”检索对应的音乐视频
• 用西班牙语描述“D大调的旋律，流畅的和声进程”找到“Canon In D Major”
• 通过中文查询“D大调四四拍的爱尔兰舞曲”找到相关的爱尔兰音乐
• 使用希腊语查询“具有精神氛围的圣歌”，找到合适的音乐
• 将图像如“圣诞树在港口的夜景”转化为音乐检索结果

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages价格

CLaMP 3目前提供免费在线演示，用户可以通过其Hugging Face空间直接体验。关于API和企业级服务的具体价格，需联系CLaMP 3团队获取更多定价信息。

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages公司名称

CLaMP 3由多所知名学术机构联合研发，包括中央音乐学院、香港科技大学、纽约大学上海校区、穆罕默德·本·扎耶德人工智能大学、韩国高等科学技术学院及清华大学。

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages联系方式

CLaMP 3的Email地址：[[email protected]](mailto:[email protected])

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages社交媒体

CLaMP 3的社交媒体： - Twitter: @CLaMP3_MIR - Instagram: @clamp3_music