`it d dl mZ d dlmZmZ d dlZd dlZddlm Z m
Z
ddl m Z ddl
mZmZ e
j e Z e ed d
G d d e ZdS )
)UserDict)AnyUnionN )add_end_docstringslogging ) ffmpeg_read)Pipelinebuild_pipeline_init_argsT)has_feature_extractor
has_tokenizerc e Zd ZdZdZdZdZdZ fdZde e
j e e
ef dedeee
ef f fd Zd Zdd Zd
Zd Z xZS )#ZeroShotAudioClassificationPipelinea
Zero shot audio classification pipeline using `ClapModel`. This pipeline predicts the class of an audio when you
provide an audio and a set of `candidate_labels`.
The default `hypothesis_template` is : `"This is a sound of {}."`. Make sure you update it for your usage.
Example:
```python
>>> from transformers import pipeline
>>> from datasets import load_dataset
>>> dataset = load_dataset("ashraq/esc50")
>>> audio = next(iter(dataset["train"]["audio"]))["array"]
>>> classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")
>>> classifier(audio, candidate_labels=["Sound of a dog", "Sound of vacuum cleaner"])
[{'score': 0.9996, 'label': 'Sound of a dog'}, {'score': 0.0004, 'label': 'Sound of vacuum cleaner'}]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This audio
classification pipeline can currently be loaded from [`pipeline`] using the following task identifier:
`"zero-shot-audio-classification"`. See the list of available models on
[huggingface.co/models](https://huggingface.co/models?filter=zero-shot-audio-classification).
FTc t j di | | j dk rt d| j d d S )NptzThe z is only available in PyTorch. )super__init__ framework
ValueError __class__)selfkwargsr s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/pipelines/zero_shot_audio_classification.pyr z,ZeroShotAudioClassificationPipeline.__init__D sQ ""6""" >T ! ! RDN R R RSS S " ! audiosr returnc 8 t j |fi |S )a
Assign labels to the audio(s) passed as inputs.
Args:
audios (`str`, `list[str]`, `np.array` or `list[np.array]`):
The pipeline handles three types of inputs:
- A string containing a http link pointing to an audio
- A string containing a local path to an audio
- An audio loaded in numpy
candidate_labels (`list[str]`):
The candidate labels for this audio. They will be formatted using *hypothesis_template*.
hypothesis_template (`str`, *optional*, defaults to `"This is a sound of {}"`):
The format used in conjunction with *candidate_labels* to attempt the audio classification by
replacing the placeholder with the candidate_labels. Pass "{}" if *candidate_labels* are
already formatted.
Return:
A list of dictionaries containing one entry per proposed label. Each dictionary contains the
following keys:
- **label** (`str`) -- One of the suggested *candidate_labels*.
- **score** (`float`) -- The score attributed by the model to that label. It is a value between
0 and 1, computed as the `softmax` of `logits_per_audio`.
)r __call__)r r r r s r r z,ZeroShotAudioClassificationPipeline.__call__K s$ . uww 11&111r c L i }d|v r |d |d< d|v r |d |d< |i i fS )Ncandidate_labelshypothesis_templater )r r preprocess_paramss r _sanitize_parametersz8ZeroShotAudioClassificationPipeline._sanitize_parametersd sO ' '4:;M4N 0 1 F * *7=>S7T 3 4 "b((r NThis is a sound of {}.c , t |t r| d s| d rt j | j }nzBZeroShotAudioClassificationPipeline.preprocess.. s( MMMq(//22MMMr T)r* padding text_inputs)
isinstancestr
startswithrequestsgetcontentopenreadbytesr
feature_extractorr) npndarray TypeErrorlenshaper r todtype tokenizer)r audior" r# finputs sequencesr1 s ` r
preprocessz.ZeroShotAudioClassificationPipeline.preprocessm s eS ! ! % **
%e.>.>z.J.J
% ! U++3%&& %! FFHHE% % % % % % % % % % % % % % % eU # # M t'='KLLE%,, B @AA A u{ q mnn n''
G4#9#GX\ (
>T ! !YYtz**F%5!"MMMM??"&&}55 k!nh / / ,%a.KK &a.+K $*;;{;l;; !1 .
r c * | d }|d d }| j dk r+| d }| }nt d d t t
|| d D }|S )
Nr" rJ r r )dimz `tf` framework not supported.c g | ]
\ }}||d S ))scorelabelr )r- rU candidate_labels r r/ zCZeroShotAudioClassificationPipeline.postprocess.. s4
& o 6 6
r c | d S )Nr r )r. s r zAZeroShotAudioClassificationPipeline.postprocess.. s _`ab_c^c r )key)rK r softmaxtolistr sortedzip)r rP r" rJ probsscoresresults r postprocessz/ZeroShotAudioClassificationPipeline.postprocess s (,,-?@@ x(+ >T ! !NNqN))E\\^^FF <== =
*0V=M1N1NTcTc*d*d*d
r )Nr& )__name__
__module__ __qualname____doc___load_processor_load_image_processor_load_feature_extractor_load_tokenizerr r r< r= r: r3 dictr listr r% rH rQ rb
__classcell__)r s @r r r s : O ! "OT T T T T2uRZT%A B 2c 2VZ[_`ceh`h[iVj 2 2 2 2 2 22) ) ) : " r r ) collectionsr typingr r numpyr< r5 utilsr r audio_classificationr
baser r
get_loggerrc loggerr r r r rv s ! . - - - - - 4 4 4 4 4 4 4 4
H % % ,,4W[\\\]]H H H H H( H H ^]H H Hr