Seperate AutoModel class for CTC and bug fixes for WhisperTokenizer

CodeWithKyrian · CodeWithKyrian · commit 20add8562d98 · 2024-08-14T19:31:07.000+01:00
diff --git a/src/PretrainedTokenizers/PretrainedTokenizer.php b/src/PretrainedTokenizers/PretrainedTokenizer.php
@@ -539,7 +539,6 @@ public function batchDecode(array $batch, bool $skipSpecialTokens = false, ?bool
     public function decode(array $tokenIds, bool $skipSpecialTokens = false, ?bool $cleanUpTokenizationSpaces = null): string
     {
         if (empty($tokenIds) || !is_int($tokenIds[0])) {
-            dd($tokenIds);
             throw new Exception("token_ids must be a non-empty array of integers.");
         }
 

Original file line number	Diff line number	Diff line change
`@@ -539,7 +539,6 @@ public function batchDecode(array $batch, bool $skipSpecialTokens = false, ?bool`
`539`	`539`	`public function decode(array $tokenIds, bool $skipSpecialTokens = false, ?bool $cleanUpTokenizationSpaces = null): string`
`540`	`540`	`{`
`541`	`541`	`if (empty($tokenIds) \|\| !is_int($tokenIds[0])) {`
`542`		`- dd($tokenIds);`
`543`	`542`	`throw new Exception("token_ids must be a non-empty array of integers.");`
`544`	`543`	`}`
`545`	`544`