遍布即時多媒體系統與技術---子計畫四：多型態多媒體為基礎之內容分析、索引與查詢

Abstract

近年來由於數位化的影音多媒體資料爆增，因此數位內容的分析與瞭解成為刻不容緩的 熱門研究方向。數位內容的分析與瞭解是利用多媒體資料中視訊音訊的低階特性之擷取與分 析，藉以達成數位內容中事件偵測與行為辨識的目的。本計畫的研究主題分述如下： 1. 多型態數位內容之瞭解–事件偵測與行為分析 目前數位內容瞭解的一個熱門研究方向就是事件偵測。藉由視訊資料特徵之擷取，如顏 色、材質、移動，並結合含有特殊性質之音訊資料，如驟響聲、尖叫聲、碰撞聲，設計相關 演算法來推論是否有特殊事件的發生，以提供多型態之影音索引與查詢功能。同時利用影片 的連續性與時間性，我們可以針對許多正常與不正常的行為加以辨識，這可以應用在居家看 護系統、安全監控系統、道路監控系統或是運動比賽的敵情偵探系統。本計畫將分三年依序 分別完成：(a) 視訊與音訊資料低階特徵之擷取，觀察特殊事件與各影音特徵之關聯性，以 供事件偵測與行為辨識之用。(b) 依影音特徵與相關事件行為，設計適當之演算法來偵測該 事件與辯識該行為，並設法提升準確率。(c)因應不同事件或不同的使用者需求，建構如居家 看護系統、道路監控系統或是運動比賽的敵情偵探系統…等相關之系統。 2. 音樂分析 隨著多媒體資訊的數位化與網際網路規模的不斷擴大，音樂資料已由原本的類比型態轉 換成目前網路上大量普及的數位型態。如何在這樣廣泛且龐大的音樂領域中，取得所想要的 音樂類型或是特定歌曲，是一個非常值得研究的課題。在音樂訊號中，節奏與旋律是主要的 兩大特性。節奏特性是屬於時間上的結構，拍子的週期性與規律性營造出節奏的感覺。至於 旋律，則屬於頻譜上的結構，像是音高、和音、和弦，皆是屬於旋律的特性。我們將利用這 些特性對音樂的分析，取得音樂的架構組成、音樂的節奏成分以及節拍的轉折等有用的資訊， 以求對各式各樣的音樂做正確的分類，並建構相關的應用系統，提供使用者索引與查詢。另 一方面，我們可以利用分析而得之音樂節奏成分，來設計一伴奏系統，隨著音樂的節拍，可 以產生不同的打擊樂器聲以作為伴奏。本計畫將分三年依序分別完成：(a)音訊資料低階特徵 之擷取，並推論出音樂中節奏與旋律有關之特性。(b) 設計適當之演算法來對音樂做分類， 並建構相關索引與查詢系統系統。(c) 設計音樂節奏分析之演算法，建構出一個可以跟上音樂 節拍的自動伴奏系統。

Due to the explosion of multimedia data in recent years, digital content analysis and understanding becomes a hot research topic. The goal of digital content analysis and understanding is to extract and to analyze the low level features of audio and video data for the achievement of behavior recognition and event detection. The research topics of the project are described as follows. 1. Multimodal Digital Content Understanding–Event Detection and Behavior Analysis One of the hottest research topics in digital content understanding is event detection. The features extracted from video data, such as color, texture and motion, and the some special sounds from the audio data are utilized together to infer whether some specific events take place. The information of event detection provides users the cues to efficiently index and query the digital contents. Moreover, the normal and abnormal behaviors can be analyzed by the characteristics of temporal continuity of the video, which can be applied to many applications, such as home-caring system, security surveillance system, traffic surveillance system and sports video surveillance and statistic system. As a result, the project will respectively study the following issues: (a) Extract low level features from audio and video data, and investigate the relationship between the features and the specific events for event detection and behavior analysis. (b) According to the association between the features and the specific events, design algorithms for event detection and behavior analysis, and improve the accuracy of detection and identification. (c) Develop application prototype systems, such as home-caring system, security surveillance system, traffic surveillance system and sports video surveillance and statistic system. 2. Music analysis With the digitalization of multimedia data and the explosion of Internet, music data are transformed into digital type broadcasted broadly. How to retrieve a preferred music piece among the comprehensive music domain is a hot research topic. Melody and rhythm are two major properties in music. Melody is a spectral structure containing features such as chord, harmonic and pitch. Rhythm is a temporal structure that beats correspond only to the sense of equally spaced temporal units, and the periodicity and regularity make the sense of rhythm. Exploiting these properties of music will lead to more high-level cognition of music signals, such as the organization of music, the tempo components of rhythm and the transitions of rhythm. Then, various genres of music can be classified. With a well-constructed system, users can retrieve and query efficiently. On the other hand, by exploiting the components of rhythm, an automatic accompanying system can be designed to play sounds of different percussion instruments following the music. As a result, the project will respectively study the following issues: (a) Extract low level features from audio data, and infer the two major properties in music, melody and rhythm. (b) Design algorithms for music classification, and construct an indexing and query system. (c) Design algorithm for rhythm analysis, and develop an automatic accompanying system capable of following the music.