ViperEkura
/
AstrAI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(tokenizer): 修复stop_ids属性返回错误的token ID列表

This commit is contained in:
ViperEkura 2025-10-31 19:19:38 +08:00
parent 5d3799b715
commit e86328b753
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
khaosz/data/tokenizer.py
View File

@ -93,7 +93,8 @@ class BpeTokenizer:
@property @property
def stop_ids(self) -> List[int]: def stop_ids(self) -> List[int]:
stop_ids = self._control_tokens + self._special_tokens stop_token = self._control_tokens + self._special_tokens
stop_ids = [self._tokenizer.token_to_id(token) for token in stop_token]
return stop_ids return stop_ids
@property @property