ほとんどのエンジニアが知っている2017年論文「Attention is All You Need」はエンコーダ-デコーダ構造を前提に、両側にマルチヘッド self-attentionを導入したattentionベースのシーケンス変換モデルを提案します。これに対応する特許がまさに「Attention-based sequence transduction neural networks」ファミリで、エンコーダとデコーダの両方を含む典型的な構造を包括的に請求しています。
A method for generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens selected from a vocabulary that includes natural language tokens,
the method comprising, at each of a plurality of generation time steps:
generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step;
processing the combined sequence using a self-attention decoder neural network that comprises a plurality of masked self-attention neural network layers, and wherein the self-attention decoder neural network is configured to process the combined sequence to generate a time step output that defines a score distribution over a set of possible output tokens,
wherein the masked self-attention neural network layers are masked such that the time step output depends only on the input sequence and the output tokens that have already been generated as of the generation time step and not on any output tokens that are after the last token that had already been generated in the output sequence; and
determining an output token using the time step output.
US 12,271,817 B2では、入力トークンを含む語彙が自然言語トークンを含むvocabularyであることを指定し、自然言語LLMに直接対応する形式に範囲を調整します。同じ技術思想を維持しながらも、実際のチャットボット・テキスト生成サービスに対する適用可能性をより鮮明にするための選択と解釈できます。
[4] US 12,299,572 B2 (Mixture-of-Expertsを含む)
Claim 1
A method for generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens,
the method comprising, at each of a plurality of generation time steps:
generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step;
processing the combined sequence using a self-attention decoder neural network that comprises (i) one or more masked self-attention neural network layers and (ii) one or more mixture of expert layers to generate a time step output that defines a score distribution over a set of possible output tokens,
wherein the masked self-attention neural network layers are masked such that the time step output depends only on the input sequence and the output tokens that have already been generated as of the generation time step and not on any output tokens that are after the last token that had already been generated in the output sequence; and
determining an output token using the time step output.
US 12,299,572 B2は、デコーダ内にMixture-of-Experts（MoE）レイヤを明示的に含めることによって、最近の大規模モデルで広く使用されているMoEスケーリング技術を組み合わせたバージョンです。 self-attentionレイヤーとMoEレイヤーを一緒に使用するデコーダー構造を実装する場合は、この特許との衝突の可能性まで別途検討する必要があります。
[5] US 12,299,573 B2（画像生成版）
Claim 1
A method for processing an input sequence comprising a plurality of input tokens, the method comprising:
at each of a plurality of generation time steps:
generating a combined sequence for the generation time step that includes the input sequence followed by output tokens based on time step outputs that have already been generated as of the generation time step;
