Skip to content

Fix Japanese TN word spans with quoted spaces#441

Open
kiwigitops wants to merge 2 commits into
NVIDIA:mainfrom
kiwigitops:codex/ja-word-spaces-in-quotes
Open

Fix Japanese TN word spans with quoted spaces#441
kiwigitops wants to merge 2 commits into
NVIDIA:mainfrom
kiwigitops:codex/ja-word-spaces-in-quotes

Conversation

@kiwigitops

Copy link
Copy Markdown

Fixes #427.

Summary:

  • lets the Japanese word classifier consume spaces inside plain word spans by matching the existing Chinese TN word behavior
  • keeps the quoted English phrase in the reported Japanese sentence verbalizable instead of leaving a failed lattice
  • adds a Japanese word regression for the “Hello World” case

Verification:

  • python -m compileall nemo_text_processing/text_normalization/ja/taggers/word.py tests/nemo_text_processing/ja/test_word.py
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failed to transfer when there is double quotations inside the input text

1 participant