{"id":26433,"date":"2024-02-25T18:35:37","date_gmt":"2024-02-25T19:35:37","guid":{"rendered":"https:\/\/express24.ir\/d\/product\/%d9%85%d9%82%d8%a7%d9%84%d9%87-%d8%a7%d9%84%da%af%d9%88%d8%b1%db%8c%d8%aa%d9%85-%db%8c%da%a9-%d8%ac%d8%b1%db%8c%d8%a7%d9%86-%d8%b9%d8%a8%d9%88%d8%b1-%d8%a8%d8%b1%d8%a7%db%8c-%d8%aa%d9%82%d8%b1%db%8c\/"},"modified":"2024-02-25T18:35:38","modified_gmt":"2024-02-25T19:35:38","slug":"%d9%85%d9%82%d8%a7%d9%84%d9%87-%d8%a7%d9%84%da%af%d9%88%d8%b1%db%8c%d8%aa%d9%85-%db%8c%da%a9-%d8%ac%d8%b1%db%8c%d8%a7%d9%86-%d8%b9%d8%a8%d9%88%d8%b1-%d8%a8%d8%b1%d8%a7%db%8c-%d8%aa%d9%82%d8%b1%db%8c","status":"publish","type":"product","link":"https:\/\/express24.ir\/d\/product\/%d9%85%d9%82%d8%a7%d9%84%d9%87-%d8%a7%d9%84%da%af%d9%88%d8%b1%db%8c%d8%aa%d9%85-%db%8c%da%a9-%d8%ac%d8%b1%db%8c%d8%a7%d9%86-%d8%b9%d8%a8%d9%88%d8%b1-%d8%a8%d8%b1%d8%a7%db%8c-%d8%aa%d9%82%d8%b1%db%8c\/","title":{"rendered":"\u0645\u0642\u0627\u0644\u0647 \u0627\u0644\u06af\u0648\u0631\u06cc\u062a\u0645 \u06cc\u06a9 \u062c\u0631\u06cc\u0627\u0646 \u0639\u0628\u0648\u0631 \u0628\u0631\u0627\u06cc \u062a\u0642\u0631\u06cc\u0628 \u062a\u0648\u06a9\u0646 \u0641\u0648\u0642 \u0627\u0644\u0639\u0627\u062f\u0647 \u0637\u0648\u0644\u0627\u0646\u06cc \u062f\u0631 \u0641\u0636\u0627\u06cc \u0632\u06cc\u0631 \u062e\u0637\u06cc"},"content":{"rendered":"\n\n\n\n\n\n\n\n\n\n\n
\u0639\u0646\u0648\u0627\u0646 \u0645\u0642\u0627\u0644\u0647 \u0628\u0647 \u0627\u0646\u06af\u0644\u06cc\u0633\u06cc <\/td>\nOne Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space<\/td>\n<\/tr>\n
\u0639\u0646\u0648\u0627\u0646 \u0645\u0642\u0627\u0644\u0647 \u0628\u0647 \u0641\u0627\u0631\u0633\u06cc <\/td>\n\u0645\u0642\u0627\u0644\u0647 \u0627\u0644\u06af\u0648\u0631\u06cc\u062a\u0645 \u06cc\u06a9 \u062c\u0631\u06cc\u0627\u0646 \u0639\u0628\u0648\u0631 \u0628\u0631\u0627\u06cc \u062a\u0642\u0631\u06cc\u0628 \u062a\u0648\u06a9\u0646 \u0641\u0648\u0642 \u0627\u0644\u0639\u0627\u062f\u0647 \u0637\u0648\u0644\u0627\u0646\u06cc \u062f\u0631 \u0641\u0636\u0627\u06cc \u0632\u06cc\u0631 \u062e\u0637\u06cc<\/td>\n<\/tr>\n
\u0646\u0648\u06cc\u0633\u0646\u062f\u06af\u0627\u0646 <\/td>\nRaghav Addanki, Chenyang Li, Zhao Song, Chiwun Yang<\/td>\n<\/tr>\n
\u0632\u0628\u0627\u0646 \u0645\u0642\u0627\u0644\u0647 <\/td>\n\u0627\u0646\u06af\u0644\u06cc\u0633\u06cc<\/td>\n<\/tr>\n
\u0641\u0631\u0645\u062a \u0645\u0642\u0627\u0644\u0647: <\/td>\nPDF<\/td>\n<\/tr>\n
\u062a\u0639\u062f\u0627\u062f \u0635\u0641\u062d\u0627\u062a<\/td>\n19<\/td>\n<\/tr>\n
\u062f\u0633\u062a\u0647 \u0628\u0646\u062f\u06cc \u0645\u0648\u0636\u0648\u0639\u0627\u062a <\/td>\nMachine Learning,Computation and Language,Machine Learning,\u06cc\u0627\u062f\u06af\u06cc\u0631\u06cc \u0645\u0627\u0634\u06cc\u0646 , \u0645\u062d\u0627\u0633\u0628\u0647 \u0648 \u0632\u0628\u0627\u0646 , \u06cc\u0627\u062f\u06af\u06cc\u0631\u06cc \u0645\u0627\u0634\u06cc\u0646 ,<\/td>\n<\/tr>\n
\u062a\u0648\u0636\u06cc\u062d\u0627\u062a <\/td>\nSubmitted 24 November, 2023; originally announced November 2023.<\/td>\n<\/tr>\n
\u062a\u0648\u0636\u06cc\u062d\u0627\u062a \u0628\u0647 \u0641\u0627\u0631\u0633\u06cc <\/td>\n\u0627\u0631\u0633\u0627\u0644 \u0634\u062f\u0647 24 \u0646\u0648\u0627\u0645\u0628\u0631 2023 \u061b\u062f\u0631 \u0627\u0628\u062a\u062f\u0627 \u0646\u0648\u0627\u0645\u0628\u0631 2023 \u0627\u0639\u0644\u0627\u0645 \u0634\u062f.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u0686\u06a9\u06cc\u062f\u0647<\/h2>\n

Deploying Large Language Models (LLMs) in streaming applications that involve long contexts, particularly for extended dialogues and text analysis, is of paramount importance but presents two significant challenges. Firstly, the memory consumption is substantial during the decoding phase due to the caching of Key and Value states (KV) of previous tokens. Secondly, attention computation is time-consuming with a time complexity of $O(n^2)$ for the generation of each token. In recent OpenAI DevDay (Nov 6, 2023), OpenAI released a new model that is able to support a 128K-long document, in our paper, we focus on the memory-efficient issue when context length $n$ is much greater than 128K ($n \\gg 2^d$). Considering a single-layer self-attention with Query, Key, and Value matrices $Q, K, V \\in \\mathbb{R}^{n \\times d}$, the polynomial method approximates the attention output $T \\in \\mathbb{R}^{n \\times d}$. It accomplishes this by constructing $U_1, U_2 \\in \\mathbb{R}^{n \\times t}$ to expedite attention ${\\sf Attn}(Q, K, V)$ computation within $n^{1+o(1)}$ time executions. Despite this, storing the Key and Value matrices $K, V \\in \\mathbb{R}^{n \\times d}$ still necessitates $O( n d)$ space, leading to significant memory usage. In response to these challenges, we introduce a new algorithm that only reads one pass of the data in streaming fashion. This method employs sublinear space $o(n)$ to store three sketch matrices, alleviating the need for exact $K, V$ storage. Notably, our algorithm exhibits exceptional memory-efficient performance with super-long tokens. As the token length $n$ increases, our error guarantee diminishes while the memory usage remains nearly constant. This unique attribute underscores the potential of our technique in efficiently handling LLMs in streaming applications.<\/p>\n

\u0686\u06a9\u06cc\u062f\u0647 \u0628\u0647 \u0641\u0627\u0631\u0633\u06cc (\u062a\u0631\u062c\u0645\u0647 \u0645\u0627\u0634\u06cc\u0646\u06cc)<\/h2>\n

\u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0627\u0632 \u0645\u062f\u0644 \u0647\u0627\u06cc \u0628\u0632\u0631\u06af \u0632\u0628\u0627\u0646 (LLM) \u062f\u0631 \u0628\u0631\u0646\u0627\u0645\u0647 \u0647\u0627\u06cc \u062c\u0631\u06cc\u0627\u0646 \u06a9\u0647 \u0634\u0627\u0645\u0644 \u0632\u0645\u06cc\u0646\u0647 \u0647\u0627\u06cc \u0637\u0648\u0644\u0627\u0646\u06cc \u060c \u0628\u0647 \u0648\u06cc\u0698\u0647 \u0628\u0631\u0627\u06cc \u062f\u06cc\u0627\u0644\u0648\u06af \u0647\u0627\u06cc \u06af\u0633\u062a\u0631\u062f\u0647 \u0648 \u062a\u062c\u0632\u06cc\u0647 \u0648 \u062a\u062d\u0644\u06cc\u0644 \u0645\u062a\u0646 \u060c \u0627\u0632 \u0627\u0647\u0645\u06cc\u062a \u0648\u06cc\u0698\u0647 \u0627\u06cc \u0628\u0631\u062e\u0648\u0631\u062f\u0627\u0631 \u0627\u0633\u062a \u0627\u0645\u0627 \u062f\u0648 \u0686\u0627\u0644\u0634 \u0645\u0647\u0645 \u0631\u0627 \u0627\u0631\u0627\u0626\u0647 \u0645\u06cc \u062f\u0647\u062f.\u062f\u0631 \u0645\u0631\u062d\u0644\u0647 \u0627\u0648\u0644 \u060c \u0645\u0635\u0631\u0641 \u062d\u0627\u0641\u0638\u0647 \u062f\u0631 \u0645\u0631\u062d\u0644\u0647 \u0631\u0645\u0632\u06af\u0634\u0627\u06cc\u06cc \u0628\u0647 \u062f\u0644\u06cc\u0644 \u0630\u062e\u06cc\u0631\u0647 \u062d\u0627\u0644\u062a \u0647\u0627\u06cc \u06a9\u0644\u06cc\u062f\u06cc \u0648 \u0627\u0631\u0632\u0634 (kV) \u0646\u0634\u0627\u0646\u0647 \u0647\u0627\u06cc \u0642\u0628\u0644\u06cc \u0642\u0627\u0628\u0644 \u062a\u0648\u062c\u0647 \u0627\u0633\u062a.\u062b\u0627\u0646\u06cc\u0627 \u060c \u0645\u062d\u0627\u0633\u0628\u0627\u062a \u062a\u0648\u062c\u0647 \u0628\u0627 \u067e\u06cc\u0686\u06cc\u062f\u06af\u06cc \u0632\u0645\u0627\u0646\u06cc $ O (n^2) $ \u0628\u0631\u0627\u06cc \u062a\u0648\u0644\u06cc\u062f \u0647\u0631 \u0646\u0634\u0627\u0646\u0647 \u0648\u0642\u062a \u06af\u06cc\u0631 \u0627\u0633\u062a.\u062f\u0631 Openai Devday \u0627\u062e\u06cc\u0631 (6 \u0646\u0648\u0627\u0645\u0628\u0631 2023) \u060c OpenAI \u0645\u062f\u0644 \u062c\u062f\u06cc\u062f\u06cc \u0631\u0627 \u0645\u0646\u062a\u0634\u0631 \u06a9\u0631\u062f \u06a9\u0647 \u0642\u0627\u062f\u0631 \u0628\u0647 \u067e\u0634\u062a\u06cc\u0628\u0627\u0646\u06cc \u0627\u0632 \u06cc\u06a9 \u0633\u0646\u062f \u0628\u0647 \u0637\u0648\u0644 128k \u0627\u0633\u062a \u060c \u062f\u0631 \u0645\u0642\u0627\u0644\u0647 \u0645\u0627 \u060c \u0645\u0627 \u0631\u0648\u06cc \u0645\u0633\u0626\u0644\u0647 \u06a9\u0627\u0631\u0622\u0645\u062f \u062d\u0627\u0641\u0638\u0647 \u062a\u0645\u0631\u06a9\u0632 \u0645\u06cc \u06a9\u0646\u06cc\u0645 \u0648\u0642\u062a\u06cc \u06a9\u0647 \u0637\u0648\u0644 \u0632\u0645\u06cc\u0646\u0647 $ n $ \u0628\u0633\u06cc\u0627\u0631 \u0628\u06cc\u0634\u062a\u0631 \u0627\u0632 128K \u0627\u0633\u062a.($ n gg 2^d $).\u0628\u0627 \u062f\u0631 \u0646\u0638\u0631 \u06af\u0631\u0641\u062a\u0646 \u06cc\u06a9 \u062a\u0648\u062c\u0647 \u062e\u0648\u062f \u06cc\u06a9 \u0644\u0627\u06cc\u0647 \u0628\u0627 \u0645\u0627\u062a\u0631\u06cc\u0633 \u067e\u0631\u0633 \u0648 \u062c\u0648 \u060c \u06a9\u0644\u06cc\u062f \u0648 \u0627\u0631\u0632\u0634 $ q \u060c k \u060c v in mathbb {r}^{n times d} $ \u060c \u0631\u0648\u0634 \u0686\u0646\u062f \u062c\u0645\u0644\u0647 \u0627\u06cc \u062e\u0631\u0648\u062c\u06cc \u062a\u0648\u062c\u0647 $ t \u062f\u0631 \u0631\u0627 \u062a\u0642\u0631\u06cc\u0628 \u0645\u06cc \u062f\u0647\u062f.Mathbb {r}^{n times d} $.\u0627\u06cc\u0646 \u06a9\u0627\u0631 \u0631\u0627 \u0628\u0627 \u0633\u0627\u062e\u062a $ u_1 \u060c u_2 in mathbb {r}^{n times t} $ \u0627\u0646\u062c\u0627\u0645 \u0645\u06cc \u062f\u0647\u062f \u062a\u0627 \u062a\u0648\u062c\u0647 $ { sf attn} (q \u060c k \u060c v) \u0645\u062d\u0627\u0633\u0628\u0647 $ \u062f\u0631 $ n^{1+o(1) \u0627\u0639\u062f\u0627\u0645 \u0632\u0645\u0627\u0646 $.\u0628\u0627 \u0648\u062c\u0648\u062f \u0627\u06cc\u0646 \u060c \u0630\u062e\u06cc\u0631\u0647 \u0645\u0627\u062a\u0631\u06cc\u0633 \u06a9\u0644\u06cc\u062f \u0648 \u0627\u0631\u0632\u0634 $ k \u060c v in mathbb {r}^{n times d} $ \u0647\u0646\u0648\u0632 \u0647\u0645 \u0628\u0647 \u0641\u0636\u0627\u06cc $ (n d) $ \u0646\u06cc\u0627\u0632 \u062f\u0627\u0631\u062f \u0648 \u0645\u0646\u062c\u0631 \u0628\u0647 \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0642\u0627\u0628\u0644 \u062a\u0648\u062c\u0647 \u062d\u0627\u0641\u0638\u0647 \u0645\u06cc \u0634\u0648\u062f.\u062f\u0631 \u067e\u0627\u0633\u062e \u0628\u0647 \u0627\u06cc\u0646 \u0686\u0627\u0644\u0634 \u0647\u0627 \u060c \u0645\u0627 \u06cc\u06a9 \u0627\u0644\u06af\u0648\u0631\u06cc\u062a\u0645 \u062c\u062f\u06cc\u062f \u0631\u0627 \u0645\u0639\u0631\u0641\u06cc \u0645\u06cc \u06a9\u0646\u06cc\u0645 \u06a9\u0647 \u0641\u0642\u0637 \u06cc\u06a9 \u067e\u0627\u0633 \u0627\u0632 \u062f\u0627\u062f\u0647 \u0647\u0627 \u0631\u0627 \u062f\u0631 \u0645\u062f \u062c\u0631\u06cc\u0627\u0646 \u0645\u06cc \u062e\u0648\u0627\u0646\u062f.\u0627\u06cc\u0646 \u0631\u0648\u0634 \u0627\u0632 \u0641\u0636\u0627\u06cc \u0632\u06cc\u0631\u0646\u0648\u06cc\u0633 $ o (n) $ \u0628\u0631\u0627\u06cc \u0630\u062e\u06cc\u0631\u0647 \u0633\u0647 \u0645\u0627\u062a\u0631\u06cc\u0633 \u0637\u0631\u062d \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0645\u06cc \u06a9\u0646\u062f \u0648 \u0646\u06cc\u0627\u0632 \u0628\u0647 \u0630\u062e\u06cc\u0631\u0647 \u062f\u0642\u06cc\u0642 $ k \u060c v $ \u0631\u0627 \u06a9\u0627\u0647\u0634 \u0645\u06cc \u062f\u0647\u062f.\u0646\u06a9\u062a\u0647 \u0642\u0627\u0628\u0644 \u062a\u0648\u062c\u0647 \u060c \u0627\u0644\u06af\u0648\u0631\u06cc\u062a\u0645 \u0645\u0627 \u0639\u0645\u0644\u06a9\u0631\u062f \u0627\u0633\u062a\u062b\u0646\u0627\u06cc\u06cc \u062d\u0627\u0641\u0638\u0647 \u0628\u0627 \u0646\u0634\u0627\u0646\u0647 \u0647\u0627\u06cc \u0641\u0648\u0642 \u0627\u0644\u0639\u0627\u062f\u0647 \u0637\u0648\u0644\u0627\u0646\u06cc \u0631\u0627 \u0646\u0634\u0627\u0646 \u0645\u06cc \u062f\u0647\u062f.\u0628\u0627 \u0627\u0641\u0632\u0627\u06cc\u0634 \u0637\u0648\u0644 \u062a\u0648\u06a9\u0646 $ n $ \u060c \u062e\u0637\u0627\u06cc \u0645\u0627 \u062a\u0636\u0645\u06cc\u0646 \u0645\u06cc \u0634\u0648\u062f \u062f\u0631 \u062d\u0627\u0644\u06cc \u06a9\u0647 \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0627\u0632 \u062d\u0627\u0641\u0638\u0647 \u062a\u0642\u0631\u06cc\u0628\u0627\u064b \u062b\u0627\u0628\u062a \u0627\u0633\u062a.\u0627\u06cc\u0646 \u0648\u06cc\u0698\u06af\u06cc \u0645\u0646\u062d\u0635\u0631 \u0628\u0647 \u0641\u0631\u062f \u060c \u067e\u062a\u0627\u0646\u0633\u06cc\u0644 \u062a\u06a9\u0646\u06cc\u06a9 \u0645\u0627 \u0631\u0627 \u062f\u0631 \u06a9\u0627\u0631\u0622\u0645\u062f \u0628\u0627 LLM \u0647\u0627 \u062f\u0631 \u0628\u0631\u0646\u0627\u0645\u0647 \u0647\u0627\u06cc \u062c\u0631\u06cc\u0627\u0646 \u062a\u0623\u06a9\u06cc\u062f \u0645\u06cc \u06a9\u0646\u062f.<\/p>\n

\r\n\r\n\r\n
\r\n\r\n

\ud83d\udcda \u0645\u062d\u062a\u0648\u0627\u06cc \u0627\u06cc\u0646 \u0645\u062d\u0635\u0648\u0644 \u0622\u0645\u0648\u0632\u0634\u06cc (\u067e\u06a9\u06cc\u062c \u06a9\u0627\u0645\u0644)<\/h2>\r\n\r\n

\r\n\u0639\u0644\u0627\u0648\u0647 \u0628\u0631 \u0645\u0642\u0627\u0644\u0647 \u0627\u0635\u0644\u06cc \u0627\u0646\u06af\u0644\u06cc\u0633\u06cc \u06a9\u0647 \u062f\u0631\u06cc\u0627\u0641\u062a \u0645\u06cc \u06a9\u0646\u06cc\u062f\u060c \u0628\u0631\u0627\u06cc \u06cc\u0627\u062f\u06af\u06cc\u0631\u06cc \u0639\u0645\u06cc\u0642\u200c\u062a\u0631 \u0648 \u062a\u0633\u0644\u0637 \u06a9\u0627\u0645\u0644 \u0628\u0631 \u0645\u0628\u0627\u062d\u062b<\/strong> \u0645\u062c\u0645\u0648\u0639\u0647\u200c\u0627\u06cc \u0627\u0632 \u06a9\u062a\u0627\u0628\u200c\u0647\u0627\u06cc \u0622\u0645\u0648\u0632\u0634\u06cc \u0646\u06cc\u0632 \u0627\u0631\u0627\u0626\u0647 \u0645\u06cc\u200c\u0634\u0648\u062f.\r\n<\/p>\r\n\r\n