I learned Git from an O'Reilly book and I loved that it started with the internals first. The git CLI has some rough edges, but once you have concepts of work tree, index, commits and diffs down, it is extremely…
One point: if you are re-reviewing, other platforms (e.g. Phabricator, Gerrit) have much more developed ways to compare changes relative to one another.
One thing to bear in mind is that these embedding vectors are high dimensional, so that it is entirely possible that the token embedding and position embedding are near-orthogonal to one another. As a result,…
The tokenization is done by the tokenizer which can be thought of as just a function that maps strings to integers before the neural network. Tokenizers can be hand-specified or learned, but in either case this is…
This is a great set of comments/questions! To try and answer this a bit briefly: The input string is tokenized into a sequence of token indices (integers) as the first step of processing the input. For example, "Hello…
At the end of last year I put together a repository to try and show what is achieved by self-attention on a toy example: detect whether a sequence of characters contains both "a" and "b". The toy problem is useful…
I put together a repository at the end of last year to walk through a basic use of a single layer Transformer: detect whether "a" and "b" are in a sequence of characters. Everything is reproducible, so hopefully helpful…
I learned Git from an O'Reilly book and I loved that it started with the internals first. The git CLI has some rough edges, but once you have concepts of work tree, index, commits and diffs down, it is extremely…
One point: if you are re-reviewing, other platforms (e.g. Phabricator, Gerrit) have much more developed ways to compare changes relative to one another.
One thing to bear in mind is that these embedding vectors are high dimensional, so that it is entirely possible that the token embedding and position embedding are near-orthogonal to one another. As a result,…
The tokenization is done by the tokenizer which can be thought of as just a function that maps strings to integers before the neural network. Tokenizers can be hand-specified or learned, but in either case this is…
This is a great set of comments/questions! To try and answer this a bit briefly: The input string is tokenized into a sequence of token indices (integers) as the first step of processing the input. For example, "Hello…
At the end of last year I put together a repository to try and show what is achieved by self-attention on a toy example: detect whether a sequence of characters contains both "a" and "b". The toy problem is useful…
I put together a repository at the end of last year to walk through a basic use of a single layer Transformer: detect whether "a" and "b" are in a sequence of characters. Everything is reproducible, so hopefully helpful…