Deserved. word2vec is a beautiful peice of science. The hypothesis behind it was so amazing. I kinda feel like everything else in LLMs (including the attention mechanism itself) is just engineering around the representation learning breakthrough that word2vec demonstrated
And with applications far outside the original scope, which I think is one of the hallmarks of foundational discoveries. As soon as you have that one puzzle piece a whole lot of other stuff comes into focus.
I think when I saw word2vec I realized that semantic meaning can be encoded in machines just from prediction. So then when seq2seq etc came around I saw that this encoding could be used to make increasingly more convincing imitations of thought and language.
I think what wasnt obvious was when Transformers came around it wasn’t obvious that generating text could get comparable results to thinking. So when Chain of Thought etc came around it was really surprising but ReAct arch made it obvious that some control loops would be the next stage.
I think now working with big multi agent systems with Opus 5.5 its clear once we develop the right mainframe-y sorta infra (virtual filesystems, deeper tool permissions, file locking, agent metrics, etc) there is a clear path emerging to what the future of the industry will look like (imo).
Indeed. I remember taking a machine learning course as an undergrad in 2018 and scratching my head how could systems like these possibly scale to meaningfully interacting with computers let alone the real world. Well, here we are :)
The idea that a word is defined by the company it keeps was decades old. Mikolov’s insight was that you could learn it at scale with a very simple model, and that meaning would fall out as geometry. Obvious in hindsight, foundational in practice.
Not to downplay his accomplishments (I have sung my praises in another comment), but even vector models were already around and somewhat popular before. Much of it started in Information Retrieval with things like TF-IDF vectors and later PPMI matrices that were converted to small, dense matrices with SVD. Of course, later Yoshua Bengio et al. introduced neural language models for training embeddings.
There is even a paper by Levy & Goldberg from 2014 that shows that skip-gram with negative sampling implicitly factorizes a PPMI word-context matrix.
But Mikolov's formulation is very nice in that it is a simple linear model that allows for a very simple Hogwild optimization that worked very well/fast on multi-core CPUs of the time.
Well-deserved. Even though more complex language models won in the end, his stripping down of word embedding training from RNNs/convolution models to a simple linear model was the thing that was needed for compute at the time and completely dominated the field for five years or so. I do miss the times a little where you could train a cutting-edge model with a reasonable CPU (for training embeddings with word2vec) and a single GPU (some BiLSTM or transformers before BERT).
If I remember correctly, I read somewhere he initially had a hard time getting the word2vec paper accepted at a conference.
I still in awe how could someone understand well in math, is it pure nature or nurture. Could any average possible to grasp the beauty of math, starting from high school level math understanding.
His team is doing amazing work in making open weight models even better. Chinese models are notorious for their thinking verbosity and tenacity to call tools repeatedly [1] so his team is making them more efficient and on-point [2].
Deserved. word2vec is a beautiful peice of science. The hypothesis behind it was so amazing. I kinda feel like everything else in LLMs (including the attention mechanism itself) is just engineering around the representation learning breakthrough that word2vec demonstrated
And with applications far outside the original scope, which I think is one of the hallmarks of foundational discoveries. As soon as you have that one puzzle piece a whole lot of other stuff comes into focus.
I remember being presented Word2Vec as an undergrad in 2016 and I could not predict the sci-fi reality that is today
I think when I saw word2vec I realized that semantic meaning can be encoded in machines just from prediction. So then when seq2seq etc came around I saw that this encoding could be used to make increasingly more convincing imitations of thought and language.
I think what wasnt obvious was when Transformers came around it wasn’t obvious that generating text could get comparable results to thinking. So when Chain of Thought etc came around it was really surprising but ReAct arch made it obvious that some control loops would be the next stage.
I think now working with big multi agent systems with Opus 5.5 its clear once we develop the right mainframe-y sorta infra (virtual filesystems, deeper tool permissions, file locking, agent metrics, etc) there is a clear path emerging to what the future of the industry will look like (imo).
Do you think this chain will lead to AGI?
Indeed. I remember taking a machine learning course as an undergrad in 2018 and scratching my head how could systems like these possibly scale to meaningfully interacting with computers let alone the real world. Well, here we are :)
The idea that a word is defined by the company it keeps was decades old. Mikolov’s insight was that you could learn it at scale with a very simple model, and that meaning would fall out as geometry. Obvious in hindsight, foundational in practice.
Not to downplay his accomplishments (I have sung my praises in another comment), but even vector models were already around and somewhat popular before. Much of it started in Information Retrieval with things like TF-IDF vectors and later PPMI matrices that were converted to small, dense matrices with SVD. Of course, later Yoshua Bengio et al. introduced neural language models for training embeddings.
There is even a paper by Levy & Goldberg from 2014 that shows that skip-gram with negative sampling implicitly factorizes a PPMI word-context matrix.
But Mikolov's formulation is very nice in that it is a simple linear model that allows for a very simple Hogwild optimization that worked very well/fast on multi-core CPUs of the time.
Just FYI: You sang his praises. If you "sang my praises", that means you celebrated yourself.
Thanks. I misused the idiom as a non-native speaker, well try to remember next time!
Well-deserved. Even though more complex language models won in the end, his stripping down of word embedding training from RNNs/convolution models to a simple linear model was the thing that was needed for compute at the time and completely dominated the field for five years or so. I do miss the times a little where you could train a cutting-edge model with a reasonable CPU (for training embeddings with word2vec) and a single GPU (some BiLSTM or transformers before BERT).
If I remember correctly, I read somewhere he initially had a hard time getting the word2vec paper accepted at a conference.
I still in awe how could someone understand well in math, is it pure nature or nurture. Could any average possible to grasp the beauty of math, starting from high school level math understanding.
This is my first time hearing about Tomas in a while. Does anyone know what he has been up to since word2vec?
His team is doing amazing work in making open weight models even better. Chinese models are notorious for their thinking verbosity and tenacity to call tools repeatedly [1] so his team is making them more efficient and on-point [2].
[1] https://armincat.com/articles/ai-agent-executed-same-query-2...
[2] https://bottlecapai.com/post/thinkingcap-qwen3-8-flash-next/
I think I saw a few comments here from an account named "BottlecapAI". Chances are he might be actually around ; )
He's a co-founder of BottleCap AI
I met Tomas briefly in 2015, he's truly a genius.
Záchranka
Why not button for pivo.
press button
get 1 pivo
tap 2 - 4 times and get 2-4 piva
tap 5 times. get 5 piv!
So I guess he is the modern Thomas Midgley Jr., huh?