LLMs-from-scratch

mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2024-11-25 16:22:50 +08:00

Author	SHA1	Message	Date
casinca	bb31de8999	[minor] typo & comments (#441 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details * typo & comment - safe -> save - commenting code: batch_size, seq_len = in_idx.shape * comment - adding # NEW for assert num_heads % num_kv_groups == 0 * update memory wording --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2024-11-18 19:52:42 +09:00
Daniel Kleine	e95c898545	Fixed command for row 16 additional experiment (#439 ) Some checks failed Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details * fixed command for row 16 experiment * Update README.md --------- Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>	2024-11-17 06:50:00 +09:00
Sebastian Raschka	ccade77bf4	Add flexible padding bonus experiment (#438 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details * Add flexible padding bonus experiment * fix links	2024-11-15 08:51:01 +09:00
Sebastian Raschka	f6281ab91b	Add utility to prevent double execution of certain cells (#437 )	2024-11-14 19:56:49 +09:00
Sebastian Raschka	f61c008c5d	Add missing device transfer in gpt_generate.py (#436 )	2024-11-14 19:12:53 +09:00
Sebastian Raschka	27a6a7e64a	Add chapter names Some checks failed Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-11-08 08:39:34 -06:00
Sebastian Raschka	f4ed263847	Add "What's next" section (#432 ) Some checks failed Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details * Add What's next section * Delete appendix-D/01_main-chapter-code/appendix-D-Copy2.ipynb * Delete ch03/01_main-chapter-code/ch03-Copy1.ipynb * Delete appendix-D/01_main-chapter-code/appendix-D-Copy1.ipynb * Update ch07.ipynb * Update ch07.ipynb	2024-11-07 20:12:59 -06:00
rasbt	1183fd7837	add dropout scaling note Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-11-06 05:52:47 -06:00
casinca	9ce0be333b	potential little fixes `appendix-D4 .ipynb` (#427 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details * Update appendix-D.ipynb - lr missing argument for passing peak_lr to the optimizer - filling 1 step gap for gradient clipping * adjustments --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2024-11-03 12:12:58 -06:00
Sebastian Raschka	ba3137fa2c	Update CITATION.cff Some checks failed Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-11-01 21:32:17 -05:00
Sebastian Raschka	734f36aac1	Update CITATION.cff	2024-11-01 21:29:22 -05:00
rasbt	7553e87af0	Add citation file	2024-11-01 21:21:25 -05:00
rasbt	f03f545a17	Note about warm-up steps Some checks failed Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details	2024-11-01 16:47:12 -05:00
Daniel Kleine	81eed9afe2	updated RoPE statement (#423 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details * updated RoPE statement * updated .gitignore * Update ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb --------- Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>	2024-10-30 08:00:08 -05:00
Sebastian Raschka	b5f2aa3500	Update README.md Some checks failed Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-10-29 20:20:48 -05:00
ROHAN WINSOR	cd24a27161	Fix argument name in LlamaTokenizer constructor (#421 ) Some checks failed Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details This PR addresses an oversight in the LlamaTokenizer class by changing the constructor argument from filepath to tokenizer_file.	2024-10-29 18:01:36 -05:00
Daniel Kleine	e8c2f962e9	minor fixes: Llama 3.2 standalone (#420 ) Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details * minor fixes * reformat rope base as float --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2024-10-25 21:08:06 -05:00
Sebastian Raschka	1516de54a5	RoPE theta rescaling (#419 ) * rope fixes * update * update * cleanup	2024-10-25 15:27:23 -05:00
Daniel Kleine	5ff72c2850	fixed typos (#414 ) * fixed typos * fixed formatting * Update ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb * del weights after load into model --------- Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>	2024-10-24 18:23:53 -05:00
Daniel Kleine	d38083c401	Updated Llama 2 to 3 paths (#413 ) * llama 2 and 3 path fixes * updated llama 3, 3.1 and 3.2 paths * updated .gitignore * Typo fix --------- Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>	2024-10-24 07:40:08 -05:00
Sebastian Raschka	e1dfd2cb7a	Update test-requirements-extra.txt	2024-10-23 19:19:58 -05:00
Sebastian Raschka	7cd6a670ed	RoPE updates (#412 ) * RoPE updates * Apply suggestions from code review * updates * updates * updates	2024-10-23 18:07:49 -05:00
Sebastian Raschka	4f9c9fb703	Update tests.py	2024-10-23 07:48:33 -05:00
Daniel Kleine	ef4018181e	updates for PyTorch 2.5 (#408 ) * updated Dockerfile * updated MHA implementations for PT 2.5 * fixed typo * update installation instruction * Update setup/03_optional-docker-environment/.devcontainer/Dockerfile --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2024-10-22 20:23:31 -05:00
Sebastian Raschka	534a704364	RoPE increase (#407 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-10-21 19:58:38 -05:00
Sebastian Raschka	75133605c5	Set sampler in DDP example Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details	2024-10-21 09:26:01 -05:00
Sebastian Raschka	38969864e6	Add mean pooling experiment to classifier bonus experiments (#406 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.5 / test (2.5.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details * Add mean pooling experiment to classifier bonus experiments * formatting * add average embeddings option * pep8	2024-10-20 11:04:18 -05:00
Sebastian Raschka	467197bbf5	Test PyTorch 2.5 (#405 )	2024-10-20 10:23:31 -05:00
Sebastian Raschka	1f61aeb7c4	Note about SSL certificates (#404 ) Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details	2024-10-19 16:27:19 -05:00
rasbt	cd2753a36d	update mmap section Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-10-14 14:27:19 -05:00
rasbt	08362fd290	add mmap=True comparison	2024-10-14 11:09:55 -05:00
Sebastian Raschka	05b04f2a5a	Memory efficient weight loading (#401 ) * memory efficient weight loading * remove unused code	2024-10-14 10:30:25 -05:00
rasbt	a20ce1b817	remove redundant code line Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details	2024-10-13 15:58:11 -05:00
Sebastian Raschka	b6c4b2f9f1	Update bonus section formatting (#400 ) Some checks are pending Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details	2024-10-12 10:26:08 -05:00
Sebastian Raschka	233a3b0c8b	Update check-links.yml Some checks failed Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details	2024-10-11 12:20:49 -05:00
rasbt	93d9dae95f	update card	2024-10-11 12:15:01 -05:00
rasbt	1f4fca9f8e	update reference numbers	2024-10-11 12:13:10 -05:00
Sebastian Raschka	6d0f59a49c	Add MFU formula as reference material (#395 ) Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details * add MFU formula as reference material * Update previous_chapters.py	2024-10-10 19:42:53 -05:00
Sebastian Raschka	1a8d2929dd	Update check-links.yml Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-10-08 08:38:48 -05:00
Sebastian Raschka	ec18b6a8a3	Add Llama 3.2 RoPE to CI (#391 ) * add Llama 3.2 RoPE to CI * update	2024-10-08 08:28:34 -05:00
Sebastian Raschka	1eb0b3810a	Introduce buffers to improve Llama 3.2 efficiency (#389 ) * Introduce buffers to improve Llama 3.2 efficiency * update * update	2024-10-06 12:49:04 -05:00
Daniel Kleine	a0c0c765a8	fixed Llama 2 to 3.2 NBs (#388 ) * updated requirements * fixes llama2 to llama3 * fixed llama 3.2 standalone * fixed typo * fixed rope formula * Update requirements-extra.txt * Update ch05/07_gpt_to_llama/converting-llama2-to-llama3.ipynb * Update ch05/07_gpt_to_llama/converting-llama2-to-llama3.ipynb * Update ch05/07_gpt_to_llama/standalone-llama32.ipynb --------- Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>	2024-10-06 09:56:55 -05:00
Sebastian Raschka	0972ded530	Add a note about weight tying in Llama 3.2 (#386 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-10-05 09:20:54 -05:00
Sebastian Raschka	8a448a4410	Llama 3 (#384 ) * Implement Llama 3.2 * Add Llama 3.2 files * exclude IMDB link because stanford website seems down	2024-10-05 07:52:15 -05:00
Sebastian Raschka	8553644440	Llama 3.2 requirements file	2024-10-05 07:32:43 -05:00
Sebastian Raschka	b44096acef	Implement Llama 3.2 (#383 )	2024-10-05 07:30:47 -05:00
Sebastian Raschka	a5405c255d	Cos-sin fix in Llama 2 bonus notebook (#381 ) Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-10-03 20:45:40 -05:00
Sebastian Raschka	b993c2b25b	Improve rope settings for llama3 (#380 ) Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details	2024-10-03 08:29:54 -05:00
rasbt	278a50a348	add section numbers Some checks failed Code tests (Linux) / test (push) Has been cancelled Details Code tests (macOS) / test (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Has been cancelled Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Has been cancelled Details Code tests (Windows) / test (push) Has been cancelled Details Check hyperlinks / test (push) Has been cancelled Details Spell Check / spellcheck (push) Has been cancelled Details PEP8 Style checks / flake8 (push) Has been cancelled Details	2024-09-30 08:42:22 -05:00
Sebastian Raschka	4caafddb93	Improve DDP on Windows (#376 ) Some checks are pending Code tests (Linux) / test (push) Waiting to run Details Code tests (macOS) / test (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.0.1) (push) Waiting to run Details Test PyTorch 2.0 and 2.4 / test (2.4.0) (push) Waiting to run Details Code tests (Windows) / test (push) Waiting to run Details Check hyperlinks / test (push) Waiting to run Details Spell Check / spellcheck (push) Waiting to run Details PEP8 Style checks / flake8 (push) Waiting to run Details * Update DDP-script.py for Windows * Windows handling --------- Co-authored-by: Nathan Brown <nathan@nkbrown.us>	2024-09-29 16:53:48 -05:00

1 2 3 4 5 ...

755 Commits