Fix RoPE cache issue about 'find_unused_parameters' when DDP training. (#244)

* Fix issue about 'find_unused_parameters' when DDP training.

* annotation

* slim

* Fix issue about 'find_unused_parameters' when DDP training.

annotation

slim

* Update rotary_embedding_torch.py

Author: KakaruHayate

modules/commons/rotary_embedding_torch.py

@@ -306,7 +306,10 @@ def forward(
             exists(self.cached_freqs) and \
             (offset + seq_len) <= self.cached_freqs_seq_len
         ):
-            return self.cached_freqs[offset:(offset + seq_len)].detach()
+            freqs = self.cached_freqs[offset:(offset + seq_len)].detach()
+            # Fix issue about 'find_unused_parameters' when DDP training.(#244)
+            freqs = freqs + 0. * self.freqs.sum()
+            return freqs
 
         freqs = self.freqs