LLMLearn

Refer to caption

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

(1)

Refer to caption

Refer to caption

	$\displaystyle\mathrm{MultiHead}(Q,K,V)$	$\displaystyle=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}$
	$\displaystyle\text{where}~{}\mathrm{head_{i}}$	$\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})$

\mathrm{FFN}(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}

(2)

	$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$
	$\displaystyle PE_{(pos,2i+1)}=cos(pos/10000^{2i/d_{\text{model}}})$



	$O(n^{2}\cdot d)$	$O(1)$	$O(1)$
	$O(n\cdot d^{2})$	$O(n)$	$O(n)$
	$O(k\cdot n\cdot d^{2})$	$O(1)$	$O(log_{k}(n))$
	$O(r\cdot n\cdot d)$	$O(1)$	$O(n/r)$

lrate=d_{\text{model}}^{-0.5}\cdot\min({step\_num}^{-0.5},{step\_num}\cdot{% warmup\_steps}^{-1.5})

(3)

Model	BLEU
Model	EN-DE	EN-FR		EN-DE	EN-FR
ByteNet [18]	23.75
		39.2			$1.0\cdot 10^{20}$
	24.6	39.92		$2.3\cdot 10^{19}$	$1.4\cdot 10^{20}$
ConvS2S [9]	25.16	40.46		$9.6\cdot 10^{18}$	$1.5\cdot 10^{20}$
MoE [32]	26.03	40.56		$2.0\cdot 10^{19}$	$1.2\cdot 10^{20}$
		40.4			$8.0\cdot 10^{20}$
	26.30	41.16		$1.8\cdot 10^{20}$	$1.1\cdot 10^{21}$
	26.36	41.29		$7.7\cdot 10^{19}$	$1.2\cdot 10^{21}$
	27.3	38.1		$3.3\cdot 10^{18}$
	28.4	41.8		$2.3\cdot 10^{19}$

.

	$N$	$d_{\text{model}}$	$d_{\text{ff}}$	$h$	$d_{k}$	$d_{v}$	$P_{drop}$	$\epsilon_{ls}$		PPL	BLEU
	$N$	$d_{\text{model}}$	$d_{\text{ff}}$	$h$	$d_{k}$	$d_{v}$	$P_{drop}$	$\epsilon_{ls}$				$\times 10^{6}$
	6	512	2048	8	64	64	0.1	0.1	100K	4.92	25.8	65
(A)				1	512	512				5.29	24.9
				4	128	128				5.00	25.5
				16	32	32				4.91	25.8
				32	16	16				5.01	25.4
(B)					16					5.16	25.1	58
(B)					32					5.01	25.4	60
(C)	2									6.11	23.7	36
	4									5.19	25.3	50
	8									4.88	25.5	80
		256			32	32				5.75	24.5	28
		1024			128	128				4.66	26.0	168
			1024							5.12	25.4	53
			4096							4.75	26.2	90
(D)							0.0			5.77	24.6
							0.2			4.95	25.5
								0.0		4.67	25.3
								0.2		5.47	25.7
(E)										4.92	25.7
	6	1024	4096	16			0.3		300K	4.33	26.4	213


		88.3
		90.4
		90.4
		91.7
		91.3
		91.3
		91.3
		92.1
		92.1
		92.7
		93.0
		93.3

Refer to caption

Refer to caption

Refer to caption

Refer to caption

Refer to caption