Initial Chunking #14321

nune-tadevosyan · 2025-07-24T12:06:41Z

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Allows dynamic chunking mechanism for AED models.

Collection: [ASR]

Changelog

Add dynamic chunking ability .transcribe()
Support dynamic chunking with timestamps
Add unit tests

Usage

The dynamic chunking feature is automatically enabled when calling .transcribe() on a single audio file, or when using batch_size=1 with multiple audio files that are longer than 40 seconds.

from nemo.collections.asr.models import EncDecMultiTaskModel
canary_model = EncDecMultiTaskModel.from_pretrained('nvidia/canary-1b-v2')
output=canary_model.transcribe(['<file_path>'],timestamps=True) # or set to False for transcription without timestamps

print("predicted text:", output[0].text)
# word level timestamps
print(output[0].timestamp['word'])

# segment level timestamps
print(output[0].timestamp['segment'])

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

…anary2_timestamps

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

Signed-off-by: monica-sekoyan <msekoyan@nvidia.com>

…anary2_timestamps

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

Signed-off-by: monica-sekoyan <msekoyan@nvidia.com>

…anary2_timestamps

…IA/NeMo into msekoyan/canary2_timestamps

Signed-off-by: Nune <ntadevosyan@nvidia.com>

Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com>

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

…thub.com/NVIDIA/NeMo into msekoyan/canary2_timestamps

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

nemo/collections/asr/models/aed_multitask_models.py

nemo/collections/asr/parts/mixins/transcription.py

nithinraok · 2025-08-08T14:57:33Z

nemo/collections/asr/data/audio_to_text_lhotse_prompted.py

+            for i in range(audio.shape[0]):
+                waveform = audio[i, : audio_lens[i]]
+                # Split the waveform into chunks and get their lengths.
+                chunks, chunk_lens = self._chunk_waveform(waveform)


Shouldn;t be there an option for overlap control here?

nemo/collections/asr/data/audio_to_text_lhotse_prompted.py

nemo/collections/asr/models/aed_multitask_models.py

The base branch was changed.

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

github-actions · 2025-08-14T03:46:44Z

[🤖]: Hi @nune-tadevosyan 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

//cc @chtruong814 @ko3n1g @pablo-garay @thomasdhc

* adding nfa to canary Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com> * remove comments Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * modify external model loading Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * fix audio padding Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * reseting Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * handle non-possible alignment Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * add offset refinement Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * Initial Chunking Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> * Adding comments and docstrings Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> * Changes in doctrings Signed-off-by: Nune <ntadevosyan@nvidia.com> * Changes in doctrings Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> * Updates to the algrithm Signed-off-by: Nune <ntadevosyan@nvidia.com> * Update with timestamps Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> * Remove join_text Signed-off-by: Nune <ntadevosyan@nvidia.com> * Final Signed-off-by: Nune <ntadevosyan@nvidia.com> * Remove pdb Signed-off-by: Nune <ntadevosyan@nvidia.com> * Adjust timestamps Signed-off-by: Nune <ntadevosyan@nvidia.com> * Adjust timestamps Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> * Support for long audio Signed-off-by: Nune <ntadevosyan@nvidia.com> * Refactoring to keep model clean Signed-off-by: Nune <ntadevosyan@nvidia.com> * Small changes Signed-off-by: Nune <ntadevosyan@nvidia.com> * Removing changes from mixin Signed-off-by: Nune <ntadevosyan@nvidia.com> * small updates Signed-off-by: Nune <ntadevosyan@nvidia.com> * Back to main for mixin Signed-off-by: Nune <ntadevosyan@nvidia.com> * Fix for hypotheses Signed-off-by: Nune <ntadevosyan@nvidia.com> * Revert "Fix for hypotheses" This reverts commit 61fb893. Signed-off-by: Nune <ntadevosyan@nvidia.com> * Fix for hypotheses Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> * Revert "Revert "Fix for hypotheses"" This reverts commit 3c62a2d. Signed-off-by: Nune <ntadevosyan@nvidia.com> * Resolve Signed-off-by: Nune <ntadevosyan@nvidia.com> * Allowing user to control chunking Signed-off-by: Nune <ntadevosyan@nvidia.com> * Doc changes Signed-off-by: Nune <ntadevosyan@nvidia.com> * Forcing true for chunking Signed-off-by: Nune <ntadevosyan@nvidia.com> * Revert "reseting" This reverts commit 6d74ad0. Signed-off-by: monica-sekoyan <msekoyan@vidia.com> * Revert "Apply isort and black reformatting" This reverts commit 1d8c363. Signed-off-by: monica-sekoyan <msekoyan@vidia.com> * handle merge case for timestamps Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * add timestamp_type Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * add timestamps support chunked inference Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * refactor ctc timestamps to use utils Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * correct restore_token_cased with unk_token Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * use timestamps utils in rnnt_decoding Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * change external timestamps asr model loading Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * add forced aligned method tests Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * modify nfa to match new setup and utils Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * remove unused imports Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * merge conflicts Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * remove unused errors Signed-off-by: monica-sekoyan <msekoyan@vidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * remove unused import Signed-off-by: monica-sekoyan <msekoyan@vidia.com> * addressing comments, linting and flake8 Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * handle decode_ids_to_str change Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * correct usage of decode_tokens_to_str Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * update nfa docs Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * revert jupyter settings Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Merge and Tests Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * Unit tests Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * change decoding_tokens_to_str Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * change decoding_tokens_to_str Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * Update Signed-off-by: Nune <ntadevosyan@nvidia.com> * Doc updates Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * Doc updates Signed-off-by: Nune <ntadevosyan@nvidia.com> * Doc change for speech_to_text_aed_chunked_infer Signed-off-by: Nune <ntadevosyan@nvidia.com> * Remove some import Signed-off-by: Nune <ntadevosyan@nvidia.com> * Copyright Signed-off-by: Nune <ntadevosyan@nvidia.com> * Remove some import Signed-off-by: Nune <ntadevosyan@nvidia.com> * correct description Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * make private Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * rewrite restore_timestamps_asr_model Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * Update timestamps Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * Small updates Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * fix word offset logic Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * Tests update after the fix Signed-off-by: Nune <ntadevosyan@nvidia.com> * Cases for monotonicity Signed-off-by: Nune <ntadevosyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> * Tests fix Signed-off-by: Nune <ntadevosyan@nvidia.com> * Increase L0_Unit_Tests_GPU_ASR timeout to 30 Signed-off-by: Charlie Truong <chtruong@nvidia.com> --------- Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com> Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Signed-off-by: monica-sekoyan <msekoyan@nvidia.com> Signed-off-by: Nune <ntadevosyan@nvidia.com> Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Signed-off-by: monica-sekoyan <msekoyan@vidia.com> Signed-off-by: nune-tadevosyan <152167970+nune-tadevosyan@users.noreply.github.com> Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Monica Sekoyan <msekoyan@nvidia.com> Co-authored-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com> Co-authored-by: monica-sekoyan <msekoyan@vidia.com> Co-authored-by: nithinraok <nithinrao.koluguri@gmail.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Guyue Huang <guyueh@nvidia.com>

monica-sekoyan and others added 12 commits July 3, 2025 04:45

adding nfa to canary

aac4b99

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

Merge branch 'main' of https://github.com/NVIDIA/NeMo into msekoyan/c…

aac3446

…anary2_timestamps

remove comments

974356b

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

Apply isort and black reformatting

efc325d

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

modify external model loading

0692664

Signed-off-by: monica-sekoyan <msekoyan@nvidia.com>

Merge branch 'main' of https://github.com/NVIDIA/NeMo into msekoyan/c…

bfec645

…anary2_timestamps

Apply isort and black reformatting

1eb5fb9

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

fix audio padding

65225c2

Signed-off-by: monica-sekoyan <msekoyan@nvidia.com>

Merge branch 'main' of https://github.com/NVIDIA/NeMo into msekoyan/c…

4bb1693

…anary2_timestamps

Merge branch 'msekoyan/canary2_timestamps' of https://github.com/NVID…

8348c8d

…IA/NeMo into msekoyan/canary2_timestamps

reseting

6d74ad0

Signed-off-by: Nune <ntadevosyan@nvidia.com>

Apply isort and black reformatting

1d8c363

Signed-off-by: nune-tadevosyan <nune-tadevosyan@users.noreply.github.com>

github-actions bot added the ASR label Jul 24, 2025

monica-sekoyan and others added 5 commits July 27, 2025 08:48

handle non-possible alignment

737d774

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

Apply isort and black reformatting

4a862be

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

add offset refinement

27b16d7

Signed-off-by: Monica Sekoyan <msekoyan@nvidia.com>

Merge branches 'main' and 'msekoyan/canary2_timestamps' of https://gi…

65112f3

…thub.com/NVIDIA/NeMo into msekoyan/canary2_timestamps

Apply isort and black reformatting

faf407d

Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com>

nune-tadevosyan force-pushed the nune/canary_chunking branch 5 times, most recently from 5df3f2f to 90c3bec Compare August 3, 2025 13:05

nithinraok requested changes Aug 7, 2025

View reviewed changes

nune-tadevosyan force-pushed the nune/canary_chunking branch 5 times, most recently from 3dfcf67 to af3df5e Compare August 7, 2025 17:57

nithinraok reviewed Aug 8, 2025

View reviewed changes

Base automatically changed from msekoyan/canary2_timestamps to main August 13, 2025 20:27

nithinraok changed the base branch from main to msekoyan/canary2_timestamps August 13, 2025 22:01

resolve conflicts

a6f3ae0

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

github-actions bot added core Changes to NeMo Core NLP labels Aug 13, 2025

ko3n1g added Run CICD and removed Run CICD labels Aug 13, 2025

nithinraok changed the base branch from msekoyan/canary2_timestamps to main August 13, 2025 22:24

nithinraok marked this pull request as ready for review August 13, 2025 22:25

nithinraok previously approved these changes Aug 13, 2025

View reviewed changes

Increase L0_Unit_Tests_GPU_ASR timeout to 30

a984027

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 dismissed nithinraok’s stale review via a984027 August 14, 2025 00:34

chtruong814 requested review from chtruong814, ko3n1g, pablo-garay and thomasdhc as code owners August 14, 2025 00:34

github-actions bot added CI and removed core Changes to NeMo Core NLP labels Aug 14, 2025

ko3n1g added Run CICD and removed Run CICD labels Aug 14, 2025

ko3n1g temporarily deployed to test August 14, 2025 00:35 — with GitHub Actions Inactive

chtruong814 approved these changes Aug 14, 2025

View reviewed changes

chtruong814 enabled auto-merge (squash) August 14, 2025 03:32

github-actions bot removed the Run CICD label Aug 14, 2025

chtruong814 merged commit e503a6e into main Aug 14, 2025
238 of 250 checks passed

chtruong814 deleted the nune/canary_chunking branch August 14, 2025 03:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Initial Chunking #14321

Initial Chunking #14321

Uh oh!

nune-tadevosyan commented Jul 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nithinraok Aug 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Initial Chunking #14321

Initial Chunking #14321

Uh oh!

Conversation

nune-tadevosyan commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nithinraok Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nune-tadevosyan commented Jul 24, 2025 •

edited

Loading