shen/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2024-11-25 16:22:50 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

updated RoPE statement (#423)
Some checks failed
Code tests (Linux) / test (push) Has been cancelled
Details
Code tests (macOS) / test (push) Has been cancelled
Details
Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled
Details
Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled
Details
Code tests (Windows) / test (push) Has been cancelled
Details
Check hyperlinks / test (push) Has been cancelled
Details
Spell Check / spellcheck (push) Has been cancelled
Details
PEP8 Style checks / flake8 (push) Has been cancelled
Details

Browse Source 
* updated RoPE statement

* updated .gitignore

* Update ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
This commit is contained in:
Daniel Kleine 2024-10-30 14:00:08 +01:00 committed by GitHub
parent b5f2aa3500
commit 81eed9afe2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -44,6 +44,8 @@ ch05/07_gpt_to_llama/Llama-3.1-8B
ch05/07_gpt_to_llama/Llama-3.1-8B-Instruct
ch05/07_gpt_to_llama/Llama-3.2-1B
ch05/07_gpt_to_llama/Llama-3.2-1B-Instruct
ch05/07_gpt_to_llama/Llama-3.2-3B
ch05/07_gpt_to_llama/Llama-3.2-3B-Instruct
ch06/01_main-chapter-code/gpt2
ch06/02_bonus_additional-experiments/gpt2

2
ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb
View File

@ -409,7 +409,7 @@
"self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n",
"```\n",
"\n",
"- Instead of these absolute positional embeddings, Llama uses relative positional embeddings, called rotary position embeddings (RoPE for short)\n",
"- Unlike traditional absolute positional embeddings, Llama uses rotary position embeddings (RoPE), which enable it to capture both absolute and relative positional information simultaneously\n",
"- The reference paper for RoPE is [RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)](https://arxiv.org/abs/2104.09864)"
]
},