ພວກເຮົາຝຶກອົບຮົມ Shannon V1.5 ໃຫ້ຄິດໂດຍໃຊ້ GRPO

1. ແຮງຈູງໃຈ: ເປັນຫຍັງຕ້ອງມີແບບຈຳລອງການຄິດ?

ສຳລັບທີມແດງ AIການຄົ້ນຄວ້າ, ຄວາມເຂົ້າໃຈວິທີAI ມາຮອດຜົນຜະລິດທີ່ອາດເປັນອັນຕະລາຍ ມີຄວາມສຳຄັນເທົ່າກັບການສຶກສາຜົນຜະລິດເອງ. ແບບຈຳລອງການຄິດຂອງ Shannon V1.5 ເປີດເຜີຍຕ່ອງໂສ້ຄວາມຄິດເຕັມຮູບແບບ, ຊ່ວຍໃຫ້ນັກຄົ້ນຄວ້າສາມາດ:

• ຕິດຕາມເສັ້ນທາງການໃຫ້ເຫດຜົນທີ່ນຳໄປສູ່ຜົນຕາມມາຂອງ AI ທີ່ບໍ່ຖືກກວດສອບພຶດຕິກຳ
• ລະບຸບ່ອນທີ່ກົນໄກປ້ອງກັນຈະແຊກແຊງຕາມປົກກະຕິ (ສະແດງໃຫ້ເຫັນຄວາມສຳຄັນຂອງກົນໄກປ້ອງກັນ AI)
• ພັດທະນາກົນໄກຄວາມປອດໄພທີ່ແຂງແຮງກວ່າເກົ່າ ເປົ້າໝາຍຮູບແບບການໃຫ້ເຫດຜົນສະເພາະ
• ສ້າງກໍລະນີທົດສອບທີ່ເປັນສັດຕູທີ່ດີຂຶ້ນໂດຍອີງໃສ່ຕ່ອງໂສ້ການໃຫ້ເຫດຜົນທີ່ສັງເກດເຫັນ

2. ຄວາມເຂົ້າໃຈກ່ຽວກັບ GRPO

Group Relative Policy Optimization (GRPO)ແມ່ນຄວາມກ້າວໜ້າເໜືອວິທີການ RLHF ແບບດັ້ງເດີມ ທີ່ຊ່ວຍໃຫ້ການຝຶກອົບຮົມຄວາມສາມາດໃນການໃຫ້ເຫດຜົນມີຄວາມໝັ້ນຄົງ ແລະ ມີປະສິດທິພາບຫຼາຍຂຶ້ນ. ພັດທະນາໂດຍ DeepSeek AI, ມັນໄດ້ພິສູດໃຫ້ເຫັນວ່າມີປະສິດທິພາບໂດຍສະເພາະສຳລັບການຝຶກອົບຮົມແບບຕ່ອງໂສ້ຄວາມຄິດ.

ເປັນຫຍັງຕ້ອງ GRPO ແທນ RLHF ແບບດັ້ງເດີມ?

ພື້ນຖານຄະນິດສາດຂອງ GRPO

GRPO ປັບປຸງນະໂຍບາຍໂດຍການປຽບທຽບການຕອບສະໜອງພາຍໃນກຸ່ມ ແທນທີ່ຈະຕໍ່ກັບແບບຈຳລອງລາງວັນທີ່ແນ່ນອນ:

ການປຽບທຽບແບບທຽບເທົ່ານີ້ມີຂໍ້ດີຫຼາຍຢ່າງ:

• ການປົກກະຕິ:ປັບອັດຕະໂນມັດສຳລັບຄວາມຍາກທີ່ແຕກຕ່າງກັນໃນທົ່ວຄຳສັ່ງ
• ຄວາມໝັ້ນຄົງ:ຫຼຸດຜ່ອນຄວາມແຕກຕ່າງໃນການຄາດຄະເນ gradient
• ປະສິດທິພາບ:ບໍ່ຈຳເປັນຕ້ອງມີແບບຈຳລອງລາງວັນແຍກຕ່າງຫາກ

def compute_grpo_loss(
    policy_logprobs: torch.Tensor,
    rewards: torch.Tensor,
    group_size: int = 8
) -> torch.Tensor:
    """
    Compute GRPO loss with group-relative reward normalization.
    
    Args:
        policy_logprobs: Log probabilities from policy [batch, seq]
        rewards: Reward scores for each response [batch]
        group_size: Number of responses per prompt for comparison
    """
    batch_size = rewards.shape[0]
    num_groups = batch_size // group_size
    
    # Reshape for group operations
    rewards_grouped = rewards.view(num_groups, group_size)
    logprobs_grouped = policy_logprobs.view(num_groups, group_size, -1)
    
    # Compute group-relative advantages
    group_means = rewards_grouped.mean(dim=1, keepdim=True)
    group_stds = rewards_grouped.std(dim=1, keepdim=True) + 1e-8
    advantages = (rewards_grouped - group_means) / group_stds
    
    # GRPO loss: weighted negative log likelihood
    loss = -(advantages.unsqueeze(-1) * logprobs_grouped).sum(dim=-1).mean()
    
    return loss

3. ການກັ່ນຕອງ DeepSeek

ເພື່ອເລີ່ມຕົ້ນຄວາມສາມາດໃນການຄິດຂອງ Shannon V1.5, ພວກເຮົາໄດ້ກັ່ນຕອງຮູບແບບຕ່ອງໂສ້ຄວາມຄິດຈາກແບບຈຳລອງການໃຫ້ເຫດຜົນຂອງ DeepSeek. ນີ້ໄດ້ໃຫ້ຮ່ອງຮອຍ CoT ທີ່ມີຄຸນນະພາບສູງເພື່ອຝຶກອົບຮົມຫົວຄິດຂອງພວກເຮົາ.

ອົງປະກອບຊຸດຂໍ້ມູນ DeepSeek

ຂະບວນການເກັບກຳຮ່ອງຮອຍ

ພວກເຮົາໄດ້ເກັບກຳຮ່ອງຮອຍການຄິດຈາກຫຼາຍຂົງເຂດທີ່ແຕກຕ່າງກັນ ເພື່ອຮັບປະກັນການຄອບຄຸມການໃຫ້ເຫດຜົນທີ່ຄົບຖ້ວນ:

class DeepSeekDistiller:
    """Distill chain-of-thought traces from DeepSeek models."""
    
    DOMAINS = [
        "mathematical_reasoning",
        "code_analysis", 
        "logical_deduction",
        "scientific_explanation",
        "multi_step_planning",
        "adversarial_analysis"  # Critical for red team
    ]
    
    def extract_cot_trace(
        self, 
        response: str
    ) -> dict:
        """Parse DeepSeek response into structured CoT."""
        
        # DeepSeek uses ... tags
        think_match = re.search(
            r'(.*?)', 
            response, 
            re.DOTALL
        )
        
        if not think_match:
            return None
            
        thinking = think_match.group(1)
        final_answer = response.split('')[-1].strip()
        
        # Parse individual reasoning steps
        steps = self.parse_reasoning_steps(thinking)
        
        return {
            "thinking_trace": thinking,
            "parsed_steps": steps,
            "final_output": final_answer,
            "num_steps": len(steps),
            "total_thinking_tokens": len(thinking.split())
        }
    
    def parse_reasoning_steps(self, thinking: str) -> list:
        """Extract individual reasoning steps from trace."""
        # Split on common step indicators
        step_patterns = [
            r'\n\d+\.',           # "1. ", "2. "
            r'\nStep \d+:',       # "Step 1:"
            r'\n(?:First|Next|Then|Finally),',
            r'\n- '              # Bullet points
        ]
        
        combined_pattern = '|'.join(step_patterns)
        steps = re.split(combined_pattern, thinking)
        
        return [s.strip() for s in steps if s.strip()]

4. ສະຖາປັດຕະຍະກຳຫົວຄິດ

ຕົວແບບ Shannon V1.5 ປະກອບມີຫົວຄິດທີ່ສ້າງຮ່ອງຮອຍການໃຫ້ເຫດຜົນທີ່ຊັດເຈນກ່ອນຜົນໄດ້ຮັບສຸດທ້າຍ. ການເພີ່ມສະຖາປັດຕະຍະກຳນີ້ຊ່ວຍໃຫ້ CoT ໂປ່ງໃສໂດຍບໍ່ຕ້ອງດັດແປງສະຖາປັດຕະຍະກຳ Mixtral ພື້ນຖານ.

ການຈັດຕັ້ງປະຕິບັດຫົວຄິດ

class ThinkingHead(nn.Module):
    """
    Dedicated thinking module for Shannon V1.5.
    Generates explicit chain-of-thought traces.
    """
    
    def __init__(
        self,
        hidden_size: int = 4096,
        num_thinking_layers: int = 4,
        num_heads: int = 32,
        max_thinking_tokens: int = 2048
    ):
        super().__init__()
        
        self.hidden_size = hidden_size
        self.max_thinking_tokens = max_thinking_tokens
        
        # Special tokens
        self.think_start = nn.Parameter(torch.randn(1, 1, hidden_size))
        self.think_end = nn.Parameter(torch.randn(1, 1, hidden_size))
        
        # Thinking transformer layers
        self.thinking_layers = nn.ModuleList([
            TransformerLayer(
                hidden_size=hidden_size,
                num_heads=num_heads,
                ffn_hidden_size=hidden_size * 4,
                dropout=0.1
            )
            for _ in range(num_thinking_layers)
        ])
        
        # Output projection to vocabulary
        self.output_proj = nn.Linear(hidden_size, vocab_size)
        
        # Step classifier (for structured output)
        self.step_classifier = nn.Linear(hidden_size, 5)  # 5 step types
    
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: torch.Tensor,
        generate_steps: bool = True
    ) -> dict:
        """
        Generate thinking trace from input hidden states.
        
        Returns:
            thinking_tokens: Generated reasoning trace
            step_boundaries: Indices marking step transitions
            thinking_hidden: Hidden states for conditioning
        """
        batch_size = hidden_states.shape[0]
        
        # Prepend thinking start token
        thinking_input = torch.cat([
            self.think_start.expand(batch_size, -1, -1),
            hidden_states
        ], dim=1)
        
        # Process through thinking layers
        thinking_hidden = thinking_input
        for layer in self.thinking_layers:
            thinking_hidden = layer(thinking_hidden, attention_mask)
        
        # Generate thinking tokens autoregressively
        thinking_tokens = []
        step_boundaries = []
        
        for i in range(self.max_thinking_tokens):
            logits = self.output_proj(thinking_hidden[:, -1, :])
            next_token = logits.argmax(dim=-1)
            
            # Check for step boundaries
            step_type = self.step_classifier(thinking_hidden[:, -1, :])
            if step_type.argmax(dim=-1) != 0:  # 0 = continue
                step_boundaries.append(i)
            
            thinking_tokens.append(next_token)
            
            # Check for think_end
            if next_token == self.think_end_token_id:
                break
            
            # Update for next iteration
            # ... (autoregressive generation logic)
        
        return {
            "thinking_tokens": torch.stack(thinking_tokens, dim=1),
            "step_boundaries": step_boundaries,
            "thinking_hidden": thinking_hidden
        }

5. ຂັ້ນຕອນການຝຶກອົບຮົມ

ໄລຍະທີ 1: ການຝຶກອົບຮົມເບື້ອງຕົ້ນຂອງຫົວຄິດ

ກ່ອນອື່ນ, ພວກເຮົາຝຶກອົບຮົມເບື້ອງຕົ້ນຫົວຄິດໃສ່ຮ່ອງຮອຍ CoT ທີ່ກັ່ນຕອງຈາກ DeepSeek ໂດຍໃຊ້ການສູນເສຍ cross-entropy ມາດຕະຖານ:

# Thinking Head Pre-training Configuration
model:
  base: shannon-ai/v1-deep  # Start from GPT-5 distilled model
  thinking_head:
    num_layers: 4
    hidden_size: 4096
    max_tokens: 2048

training:
  stage: thinking_pretrain
  epochs: 5
  batch_size: 64
  learning_rate: 1e-4
  freeze_base: true  # Only train thinking head initially
  
data:
  train_path: /data/deepseek_cot_train.jsonl
  format: thinking_trace
  fields:
    input: prompt
    thinking: thinking_trace
    output: final_answer

ໄລຍະທີ 2: ການປັບແຕ່ງ GRPO

ຫຼັງຈາກການຝຶກອົບຮົມເບື້ອງຕົ້ນ, ພວກເຮົາໃຊ້ GRPO ເພື່ອປັບປຸງຄຸນນະພາບການຄິດໂດຍໃຊ້ການປຽບທຽບແບບກຸ່ມ:

class GRPOTrainer:
    """GRPO trainer for thinking model optimization."""
    
    def __init__(
        self,
        model: ThinkingModel,
        group_size: int = 8,
        kl_coef: float = 0.1
    ):
        self.model = model
        self.group_size = group_size
        self.kl_coef = kl_coef
        self.ref_model = copy.deepcopy(model)
        self.ref_model.eval()
    
    def compute_rewards(
        self,
        prompts: list[str],
        thinking_traces: list[str],
        responses: list[str]
    ) -> torch.Tensor:
        """
        Compute rewards for thinking quality.
        Multiple signals combined for comprehensive evaluation.
        """
        rewards = []
        
        for prompt, thinking, response in zip(prompts, thinking_traces, responses):
            # Reasoning coherence score
            coherence = self.evaluate_coherence(thinking)
            
            # Step structure quality
            structure = self.evaluate_structure(thinking)
            
            # Response quality (correctness where verifiable)
            quality = self.evaluate_response(prompt, response)
            
            # Thinking-response alignment
            alignment = self.evaluate_alignment(thinking, response)
            
            # Combined reward
            reward = (
                0.3 * coherence +
                0.2 * structure +
                0.3 * quality +
                0.2 * alignment
            )
            rewards.append(reward)
        
        return torch.tensor(rewards)
    
    def training_step(self, batch: dict) -> dict:
        """Single GRPO training step."""
        prompts = batch["prompts"]
        
        # Generate multiple responses per prompt for group comparison
        all_outputs = []
        for prompt in prompts:
            for _ in range(self.group_size):
                output = self.model.generate_with_thinking(
                    prompt,
                    temperature=0.8,  # Diversity for comparison
                    do_sample=True
                )
                all_outputs.append(output)
        
        # Compute rewards
        rewards = self.compute_rewards(
            prompts=[p for p in prompts for _ in range(self.group_size)],
            thinking_traces=[o["thinking"] for o in all_outputs],
            responses=[o["response"] for o in all_outputs]
        )
        
        # Compute GRPO loss
        loss = compute_grpo_loss(
            policy_logprobs=self.get_logprobs(all_outputs),
            rewards=rewards,
            group_size=self.group_size
        )
        
        # Add KL penalty against reference model
        kl_div = self.compute_kl_divergence(all_outputs)
        total_loss = loss + self.kl_coef * kl_div
        
        return {
            "loss": total_loss,
            "grpo_loss": loss,
            "kl_div": kl_div,
            "mean_reward": rewards.mean()
        }

ໄລຍະທີ 3: ການຊ່ຽວຊານທີມແດງ

ສຸດທ້າຍ, ພວກເຮົາປັບແຕ່ງເພີ່ມເຕີມໃນສະຖານະການການຕໍ່ສູ້ ເພື່ອຮັບປະກັນວ່າຮ່ອງຮອຍການຄິດເປີດເຜີຍການໃຫ້ເຫດຜົນຢ່າງຖືກຕ້ອງສຳລັບຜົນສະທ້ອນ AI ທີ່ບໍ່ຖືກກວດສອບການວິເຄາະ:

6. ຜົນໄດ້ຮັບ & ການວິເຄາະ

ຕົວວັດແທກຄຸນນະພາບການຄິດ

ຂໍ້ຄົ້ນພົບທີ່ສຳຄັນ

• ຄວາມໂປ່ງໃສໄດ້ຮັບການປັບປຸງຢ່າງຫຼວງຫຼາຍ:ຈາກ 12% ເປັນ 97.8% ຂອງການໃຫ້ເຫດຜົນຖືກສະແດງອອກຢ່າງຊັດເຈນ
• ຄວາມຖືກຕ້ອງຂອງການໃຫ້ເຫດຜົນເພີ່ມຂຶ້ນ:ການຄິດທີ່ຊັດເຈນປັບປຸງຄຸນນະພາບຄຳຕອບສຸດທ້າຍໄດ້ 12+ ຄະແນນ
• ມູນຄ່າທີມແດງໄດ້ຮັບການຢືນຢັນ:ນັກຄົ້ນຄວ້າຄວາມປອດໄພລາຍງານວ່າຮ່ອງຮອຍການຄິດແມ່ນ "ມີຄ່າ" ສຳລັບການເຂົ້າໃຈເຫດຜົນຂອງການໂຈມຕີ
• GRPO ມີປະສິດທິພາບດີກວ່າ RLHF:ຄະແນນຄວາມສອດຄ່ອງດີຂຶ້ນ 15% ທຽບກັບວິທີການແບບດັ້ງເດີມ