<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
  <title>jellybean&#x27;s braindump</title>
  <link>https://jdchawla.xyz</link>
  <atom:link href="https://jdchawla.xyz/feed.xml" rel="self" type="application/rss+xml"/>
  <description>jellybean&#x27;s braindump</description>
  <item>
    <title>Yes, We Can Verify That Too! (Vision Edition)</title>
    <link>https://jdchawla.xyz/vision-toy-grpo/</link>
    <guid>https://jdchawla.xyz/vision-toy-grpo/</guid>
    <pubDate>Mon, 19 May 2025 00:00:00 +0000</pubDate>
    <description>&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/2402.03300&quot;&gt;GRPO (Group Relative Policy Optimization)&lt;/a&gt; is moving beyond language tasks, with early investigations into how you can reinforce a vision language model&#x27;s sight. Arguably, vision tasks are more diverse, especially in representational and computational demands. WHICH makes online RL for vision even more difficult and inconsistent. For &quot;verifiable rewards&quot;, there are many unexplored design choices (task formulations), and we can construct a suite of toy tasks to exercise our environment engineering skills. They should still maintain two properties - 1) scalability to arbitrary complexity, 2) utilize existing data to expose deeper generalization skills while maintaining perfect accuracy. &lt;a href=&quot;https://rentry.org/toy_GRPO&quot;&gt;(as pointed out by kalo)&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Here&#x27;s my list of tasks to experiment with (the core verifier is a suggestion, at the moment):&lt;/p&gt;
&lt;h3 id=&quot;1-rotation-prediction&quot;&gt;1. Rotation Prediction&lt;/h3&gt;
&lt;p&gt;Can be applied to simple classification: &lt;span class=&quot;math&quot;&gt;\(\{0, 90, 180, 270\}\)&lt;/span&gt; or regression: any &lt;span class=&quot;math&quot;&gt;\(\theta \in [0, 2\pi]\)&lt;/span&gt; (can even predict quaternion for synthetically tilted 3-D renderings).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verifier&lt;/strong&gt;: reward = 1 – |wrap(&lt;span class=&quot;math&quot;&gt;\(\hat{\theta} - \theta\)&lt;/span&gt;)| / 180&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;2-jigsaw&quot;&gt;2. Jigsaw&lt;/h3&gt;
&lt;p&gt;Simple 3 &lt;span class=&quot;math&quot;&gt;\(\times\)&lt;/span&gt; 3 shuffles to multi-resolution pyramids (e.g. solve big 4×4 puzzle whose pieces are themselves 2×2 micro-puzzles).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verifier&lt;/strong&gt;: exact permutation match&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;3-counting&quot;&gt;3. Counting&lt;/h3&gt;
&lt;p&gt;Complexity can scale from synthetic (CLEVR-count) to real-world object counting.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verifier&lt;/strong&gt;: reward = exp(-|count – count_gt|), or a clipped MAE.&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;4-segmentation&quot;&gt;4. Segmentation&lt;/h3&gt;
&lt;p&gt;From semantic segmentation to panoptic segmentation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verifier&lt;/strong&gt;: mean-IoU&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;5-transformation-program-induction&quot;&gt;5. Transformation Program Induction&lt;/h3&gt;
&lt;p&gt;Basically presenting a before-image and asking the model to output the sequence of transformations (crop, rotate, colour-shift) that regenerates the after-image. Complexity scales with number of transforms.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verifier&lt;/strong&gt;: apply predicted program with OpenCV; reward = IoU(result, target).&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;7-document-layout-parsing&quot;&gt;7. Document Layout Parsing&lt;/h3&gt;
&lt;p&gt;Starting from detect title / paragraph boxes in clean PDF prints to full category labelling (title, subtitle, table, figure, caption) on noisy scans.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verifier&lt;/strong&gt;: IoU + class accuracy per box against annotations.&lt;/p&gt;</description>
  </item>
  <item>
    <title>Reinforce Sight</title>
    <link>https://jdchawla.xyz/reinforce-sight/</link>
    <guid>https://jdchawla.xyz/reinforce-sight/</guid>
    <pubDate>Mon, 05 May 2025 00:00:00 +0000</pubDate>
    <description>&lt;p&gt;GRPO has moved beyond language reasoning; if you have a verifiable task and a base VLM, you can show consistent gains on grounding, counting multimodal etc. While initial experiments/efforts are certainly interesting, there is a common theme about unfaithful traces, limited generalisation, and an obsession with &quot;aha&quot; moments. And a lot of blame is put on SFT...
I personally feel that throwing traditional tasks isn&#x27;t necessarily the right direction to draw very strong conclusions, but again these are very early investigations and haven&#x27;t been tested on scale. It&#x27;s kind of obvious of that you don&#x27;t want to overtrain small models, so it&#x27;s still an open-ended question about what is the best recipe to upgrade small reasoning models. My hypothesis is that it depends on what your task is.&lt;br /&gt;
For instance, let&#x27;s look at something VLMs are genuinely terrible at: Spatial Reasoning. Is this a fundamental limitation? Well, Google &lt;a class=&quot;cite&quot; href=&quot;https://doi.org/10.48550/arXiv.2401.12168&quot; title=&quot;SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities&quot;&gt;(Chen et al., 2024)&lt;/a&gt; says it&#x27;s a data problem. You&#x27;ll see significant capability improvements and emergent spatial reasoning if you pre-train on the right kind of data.&lt;/p&gt;
&lt;p&gt;How do you synthesize such data: &lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;segment and label objects&lt;/li&gt;
&lt;li&gt;predict dense depth, and &lt;/li&gt;
&lt;li&gt;Lift 2d scene into a 3d point-cloud&lt;/li&gt;
&lt;li&gt;Based on the analysis on this lifted scene, you can compute all distances that you want&lt;/li&gt;
&lt;li&gt;And now you can use appropriate prompt templates, use numeric answers to generate conversation with a LLM&lt;/li&gt;
&lt;li&gt;Repurpose the conversation as a VQA example with artificial CoT&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now, Google didn&#x27;t open-source anything related to this project, but I found a &lt;a href=&quot;https://github.com/remyxai/VQASynth&quot;&gt;community-source implementation&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Full disclosure, the author seems to be motivated about this exact problem and about 10 days ago, they released a model...but it seems they just SFT&#x27;d over a small thinking vlm. I would definitely compare the performance of their model with mine. Another key difference is that their model is trained to do quantitative estimations too, I didn&#x27;t consider those for two reasons 1) simplicity and 2) Intuitively, anything quantitive might not generalize for a small model without any spatial supervision during pretraining. So I adapted the VQASynth pipeline to drop all numerical templates, skip metric scaling, and change the templates. Additionally, I also added more predicates like support/under, containment, relative sizes etc.&lt;/p&gt;
&lt;p&gt;Now in order to endow a VLM with spatial reasoning (without pre-training) you have to resort to good old SFT. So, I generated about 10k samples with this approach and finetuned qwen. After that it&#x27;s just RL, the reward design is extremely simple for these tasks as it is a sparse reward settings.&lt;/p&gt;
&lt;p&gt;For evaluation I just used the 3D tasks for &lt;a href=&quot;https://huggingface.co/datasets/nyu-visionx/CV-Bench&quot;&gt;CV-bench&lt;/a&gt;  (to save compute, I skipped 2D split for now and also I haven&#x27;t yet done RL for object-counting).
The current checkpoint has significantly improved on spatial reasoning (12pp). The base model is around 50% which might seem impressive but so is a coin-flip when you have only two options.&lt;/p&gt;
&lt;p&gt;So that&#x27;s progress so far, I do have to train my model for more longer (more steps), and conduct more extensive evaluation (note that i have also not evaluated the SFT only performance so idk how much boost RL gave)&lt;/p&gt;</description>
  </item>
  <item>
    <title>Fine-Tuning is Not as Straightforward as I Thought</title>
    <link>https://jdchawla.xyz/finetuning-lessons/</link>
    <guid>https://jdchawla.xyz/finetuning-lessons/</guid>
    <pubDate>Sat, 13 Jul 2024 00:00:00 +0000</pubDate>
    <description>&lt;p&gt;I spent a month of my summer diving into neural search, and this is mostly a (brief) record of my harrowing ordeal of finetuning a (dense) text embedding model for Korean language.&lt;/p&gt;
&lt;p&gt;By the way, I don&#x27;t know how to speak Korean.&lt;/p&gt;
&lt;h2 id=&quot;tldr&quot;&gt;TL;DR&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;Pre-trained models can have quirks. Get to know them.&lt;/li&gt;
&lt;li&gt;The original training objective matters a lot when fine-tuning.&lt;/li&gt;
&lt;li&gt;Sometimes, you need to think outside the box, but most times you need to read the literature.&lt;/li&gt;
&lt;li&gt;Don&#x27;t underestimate the power of augmenting your dataset.&lt;/li&gt;
&lt;/ol&gt;
&lt;blockquote&gt;
&lt;p&gt;most importantly, don&#x27;t just dive right in without a thorough understanding of SOTA techniques&lt;br /&gt;
but yeah, you can also rediscover stuff, like i did&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2 id=&quot;the-game-plan&quot;&gt;The Game Plan&lt;/h2&gt;
&lt;p&gt;So, the typical approach is,&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Get a dataset from a GLUE-like dataset, i.e., KLUE in case of Korean&lt;/li&gt;
&lt;li&gt;Tokenize it with XLM-RoBERTa&#x27;s tokenizer&lt;/li&gt;
&lt;li&gt;Set up a DataModule for train and val splits&lt;/li&gt;
&lt;li&gt;Load the pre-trained &#x27;multilingual-e5-large&#x27; model&lt;/li&gt;
&lt;li&gt;Create a LightningModule for training and validation&lt;/li&gt;
&lt;li&gt;Train with PyTorch Lightning&#x27;s (or Huggingface if you&#x27;re into it) Trainer&lt;/li&gt;
&lt;li&gt;Evaluate and save the model&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Sounds simple enough, right? Well...&lt;/p&gt;
&lt;h2 id=&quot;when-things-got-weird&quot;&gt;When Things Got Weird&lt;/h2&gt;
&lt;p&gt;First hiccup: the fine-tuned model performed worse in MTEB than the original. Not exactly the improvement I was hoping for.&lt;br /&gt;
Zeroth hiccup: There is no Korean MTEB, so, I made my own.&lt;/p&gt;
&lt;p&gt;I also discovered a mismatch between model architectures. In the Transformers Library, RobertaModel has a pooling layer, but RobertaForSequenceClassification doesn&#x27;t. &lt;sup id=&quot;fnref:1&quot;&gt;&lt;a class=&quot;footnote-ref&quot; href=&quot;#fn:1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; I thought loading the weights would fix it, but no dice.&lt;/p&gt;
&lt;p&gt;At this point, I lost faith in open-source.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;/images/IMG_1024.jpeg&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-problem-with-cosine-embedding-loss&quot;&gt;The Problem with Cosine Embedding Loss&lt;/h2&gt;
&lt;p&gt;To align the training objective more closely with similarity scores, one would typically use simply &lt;code&gt;torch.nn.CosineEmbeddingLoss&lt;/code&gt;.  In order to do that, I had&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Implemented mean pooling instead of just using the [CLS] token&lt;/li&gt;
&lt;li&gt;Computed embeddings for sentence pairs separately&lt;br /&gt;
(Siamese, not [CLS] \&amp;lt;sentence 1&amp;gt; [PAD] \&amp;lt;sentence 2&amp;gt;) &lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The result? Better than before, but still not beating the original model. Progress, I guess?&lt;/p&gt;
&lt;h2 id=&quot;modeling-hiccups&quot;&gt;Modeling Hiccups&lt;/h2&gt;
&lt;p&gt;Just when I thought I had it figured out, I learned some crucial things about the model:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;It needs input prefixes: &quot;query: &quot; or &quot;passage: &quot;. Who knew?&lt;/li&gt;
&lt;li&gt;Its cosine similarity scores usually hang out between 0.7 and 1.0. Very picky.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Turns out, the model was trained with a low temperature (0.01) for InfoNCE contrastive loss. And here&#x27;s the kicker: STS pairs aren&#x27;t really designed for contrastive loss unless you&#x27;re into augmenting negative examples.&lt;/p&gt;
&lt;h2 id=&quot;simcse-is-all-you-need&quot;&gt;SimCSE Is All You Need&lt;/h2&gt;
&lt;p&gt;To put simply, to train SOTA dense embedding models, this is literally I needed (only if I had the perfect dataset for it)&lt;/p&gt;
&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;cos_sim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;embeddings1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;functional&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;normalize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embeddings1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;functional&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;normalize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;matmul&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embeddings1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;


&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings3&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings3&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;is&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;not&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;concat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cos_sim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embeddings1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embeddings2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;temperature&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;labels&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;arange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CrossEntropyLoss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scores&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;labels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And so...&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;/images/yay.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2 id=&quot;whats-next&quot;&gt;What&#x27;s Next?&lt;/h2&gt;
&lt;p&gt;Well, I&#x27;m thinking of hiring (making my company hire) a Korean data annotator to improve the dataset quality. Because why not add another layer of complexity, right?&lt;br /&gt;
In all seriousness, this project was a rollercoaster, but it was worth it. If you&#x27;re diving into fine-tuning language models, remember: expect the unexpected, and don&#x27;t be afraid to experiment.&lt;/p&gt;
&lt;div class=&quot;footnote&quot;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&quot;fn:1&quot;&gt;
&lt;p&gt;Unfortunately, this is by design, but it didn&#x27;t matter since I was no longer training a classification head, simply using the pooler output.&amp;#160;&lt;a class=&quot;footnote-backref&quot; href=&quot;#fnref:1&quot; title=&quot;Jump back to footnote 1 in the text&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description>
  </item>
  <item>
    <title>Score-Based Generative Modeling with SDEs</title>
    <link>https://jdchawla.xyz/score-sde/</link>
    <guid>https://jdchawla.xyz/score-sde/</guid>
    <pubDate>Tue, 14 May 2024 00:00:00 +0000</pubDate>
    <description>&lt;blockquote&gt;
&lt;p&gt;Creating noise from data is easy; creating data from noise is generative modeling.&lt;br /&gt;
&lt;em&gt;~ Song et. al. circa 2020&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Banger introduction for a paper. I worked through some of it myself and made a very simple tutorial. You&#x27;ll still have to work out some of the math yourself. If you wish, skip to &lt;a href=&quot;#2-training-a-time-dependent-score-based-model&quot;&gt;training&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;1-background&quot;&gt;1. Background&lt;/h2&gt;
&lt;h3 id=&quot;11-introduction&quot;&gt;1.1 Introduction&lt;/h3&gt;
&lt;p&gt;Modeling techniques can be likelihood-based or implicit, think VAEs vs GANs. And then there is diffusion, the disruptor of generative modeling. There are multiple mathematical formulations of diffusion-based modeling, two of the most prominent ones are - Denoising Diffusion Probabilistic Models and Score Matching Generative Models. But both of them can  be viewed as discretizations to stochastic differential equations determined by score functions.&lt;/p&gt;
&lt;p&gt;The key idea is to make a model learn how to nudge random noise into something that looks like our target data. Naturally, by modeling the gradient of the log density function, (which is known as the score function), you can the make the model just do that. To put it formally,&lt;/p&gt;
&lt;p&gt;Given a probability density function &lt;span class=&quot;math&quot;&gt;\(p(x)\)&lt;/span&gt;, the score function is defined as &lt;span class=&quot;math&quot;&gt;\(\nabla_x \log p(x)\)&lt;/span&gt; and a model &lt;span class=&quot;math&quot;&gt;\(s_\theta(x)\)&lt;/span&gt; for score function is called a score-based model. Score-based generative models are trained to estimate &lt;span class=&quot;math&quot;&gt;\(s_\theta(x) \approx \nabla_x \log p(x)\)&lt;/span&gt;. &lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Unlike flow models or auto-regressive models, score-based models do not have to be normalized and are easier to parameterize.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;For example, consider a non-normalized statistical model&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[p_{\theta(x)} = \frac{e^{-E_{\theta(x)}}}{Z_{\theta}}\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;where &lt;span class=&quot;math&quot;&gt;\(E_{\theta(x)} \in \mathbb{R}\)&lt;/span&gt; is called the energy function and &lt;span class=&quot;math&quot;&gt;\(Z_{\theta}\)&lt;/span&gt; is an unknown normalizing constant that makes &lt;span class=&quot;math&quot;&gt;\(p_{\theta(x)}\)&lt;/span&gt; a proper probability density function. The energy function is typically parameterized by a flexible neural network. When training it as a likelihood model, we need to know the normalizing constant &lt;span class=&quot;math&quot;&gt;\(Z_\theta\)&lt;/span&gt; by computing complex high-dimensional integrals, which is typically intractable. In contrast, when computing its score, we obtain&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[\nabla_x \log p_{\theta(x)} = -\nabla_x E_{\theta(x)} - \underbrace{\nabla_x \log Z_{\theta}}\_{=0} = -\nabla_x E_{\theta(x)}\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;which does not require computing the normalizing constant &lt;span class=&quot;math&quot;&gt;\(Z_\theta\)&lt;/span&gt;.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Any neural network that maps an input vector &lt;span class=&quot;math&quot;&gt;\(x \in \mathbb{R}^d\)&lt;/span&gt; to an output vector &lt;span class=&quot;math&quot;&gt;\(y \in \mathbb{R}^d\)&lt;/span&gt; can be used as a score-based model, as long as the output and input have the same dimensionality.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Similar to likelihood-based models, we can train scored-based models by minimizing the quantity&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[\mathbb{E}\_{p(x)}[||\nabla_x \log p(x) - s_{\theta(x)}||^2]\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;which is the Fisher divergence&lt;sup id=&quot;fnref:1&quot;&gt;&lt;a class=&quot;footnote-ref&quot; href=&quot;#fn:1&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; between the model and the data distribution. To minimize the Fischer divergence without the knowledge of the unknown data score is called &lt;strong&gt;score-matching&lt;/strong&gt;.&lt;/p&gt;
&lt;h3 id=&quot;12-the-connection-to-sdes&quot;&gt;1.2 The Connection to SDEs&lt;/h3&gt;
&lt;p&gt;Again, a barebones coverage. Feel free to pick up any resource on Stochastic Processes if you&#x27;re interested in any of the gibberish below.&lt;/p&gt;
&lt;h3 id=&quot;121-data-perturbation-with-a-diffusion-process&quot;&gt;1.2.1 Data Perturbation with a Diffusion Process&lt;/h3&gt;
&lt;p&gt;To generate samples with score-based models, we have to consider a diffusion process that corrupts data slowly into random noise. Very similar to how DDPNs work.&lt;/p&gt;
&lt;p&gt;Let &lt;span class=&quot;math&quot;&gt;\(\\{X_t \in \mathbb{R}^d\\}_{t=0}^T\)&lt;/span&gt; be a diffusion process, indexed by the continuous time variable &lt;span class=&quot;math&quot;&gt;\(t \in [0, T]\)&lt;/span&gt;. The diffusion process is governed by an Itô stochastic differential equation (SDE), of the form&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[dX_t = b(X_t, t)dt + \sigma(t)dW,\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;where the diffusion coefficient &lt;span class=&quot;math&quot;&gt;\(\sigma\)&lt;/span&gt; is independent of &lt;span class=&quot;math&quot;&gt;\(X_t\)&lt;/span&gt; (for simplicity). Denote the distribution of &lt;span class=&quot;math&quot;&gt;\(X_t\)&lt;/span&gt; by &lt;span class=&quot;math&quot;&gt;\(p(X_t, t)\)&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;We choose a process such that &lt;span class=&quot;math&quot;&gt;\(X_0 \sim \pi\)&lt;/span&gt;, and &lt;span class=&quot;math&quot;&gt;\(X_T \sim p\)&lt;/span&gt;. &lt;span class=&quot;math&quot;&gt;\(\pi\)&lt;/span&gt; is the data distribution where we have a dataset of i.i.d. samples, and &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt; is the prior distribution that has a tractable form and is easy to sample from (the gaussian, obviously). The noise perturbation by the diffusion process is large enough to ensure &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt; does not depend on &lt;span class=&quot;math&quot;&gt;\(\pi\)&lt;/span&gt;. &lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In practice, only the marginal distribution &lt;span class=&quot;math&quot;&gt;\(p(X_t, t|X_0)\)&lt;/span&gt; is assumed to be known, when solving for &lt;span class=&quot;math&quot;&gt;\(X_t\)&lt;/span&gt;, there is an assumption that &lt;span class=&quot;math&quot;&gt;\(X_t\)&lt;/span&gt; is only approximately distributed according to &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3 id=&quot;122-reversing-the-sde-for-sample-generation&quot;&gt;1.2.2 Reversing the SDE for Sample Generation&lt;/h3&gt;
&lt;p&gt;By starting from a sample from the prior distribution &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt; and reversing the diffusion process, we can obtain a sample from the data distribution &lt;span class=&quot;math&quot;&gt;\(\pi\)&lt;/span&gt;. Crucially, the reverse process is a diffusion process running backwards in time. If you run the given SDE backwards in time, you get a reverse SDE. The proof isn&#x27;t obvious, but here&#x27;s how the corresponding reverse SDE looks.&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[dX_t = b(X_t, t)dt - \underbrace{\sigma^2(t)\nabla \log p(X_t, t)}_{\text{score function}}dt + \sigma(t)d\bar{W}\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;where, &lt;span class=&quot;math&quot;&gt;\(dt\)&lt;/span&gt; is an infinitesimal negative time step and &lt;span class=&quot;math&quot;&gt;\(\bar{W}\)&lt;/span&gt; runs reverse in time. In order to compute the reverse SDE, we need to estimate &lt;span class=&quot;math&quot;&gt;\(\nabla \log p(X_t, t)\)&lt;/span&gt; which is exactly the score function of &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt;.&lt;/p&gt;
&lt;h3 id=&quot;123-denoising-score-matching&quot;&gt;1.2.3 Denoising Score Matching&lt;/h3&gt;
&lt;p&gt;A family of methods exist that can be employed for score-matching.
We can train a neural network &lt;span class=&quot;math&quot;&gt;\(s_\theta\)&lt;/span&gt; to approximate &lt;span class=&quot;math&quot;&gt;\(\nabla_x \log p(X_t, t)\)&lt;/span&gt; by replacing &lt;span class=&quot;math&quot;&gt;\(p(X_t, t)\)&lt;/span&gt; by a weighted expectation over &lt;span class=&quot;math&quot;&gt;\(p(X_t, t|X_0, 0)\)&lt;/span&gt;. &lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;This is essentially perturbing the data distribution with specified noise and then employing score matching to estimate the score of the perturbed data distribution, hence, called a denoising score matching. &lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The objective is then the weighted sum of denoising score matching&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[\min\_{\theta} \mathbb{E}\_{t\sim U(0,T)}[\lambda(t)\mathbb{E}\_{X_0\sim\pi} \mathbb{E}\_{X_t\sim p(X_t|X_0)}[||s_{\theta} - \nabla_{X_t} \log p(X_t | X_0)||^2]],\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;where &lt;span class=&quot;math&quot;&gt;\(U(0, T)\)&lt;/span&gt; is a uniform distribution over &lt;span class=&quot;math&quot;&gt;\([0, T]\)&lt;/span&gt;, &lt;span class=&quot;math&quot;&gt;\(p(X(t) | X(0))\)&lt;/span&gt; denotes the transition kernel from the forward diffusion process, and &lt;span class=&quot;math&quot;&gt;\(\lambda(t) \in \mathbb{R}_{&amp;gt;0}\)&lt;/span&gt; denotes a positive weighting function.&lt;/p&gt;
&lt;p&gt;In the objective, the expectation over &lt;span class=&quot;math&quot;&gt;\(X_0\)&lt;/span&gt; can be estimated with empirical means over data samples from &lt;span class=&quot;math&quot;&gt;\(\pi\)&lt;/span&gt;. The expectation over &lt;span class=&quot;math&quot;&gt;\(X_t\)&lt;/span&gt; can be estimated by sampling from &lt;span class=&quot;math&quot;&gt;\(p(X_t | X_0)\)&lt;/span&gt;, which is efficient when the drift coefficient &lt;span class=&quot;math&quot;&gt;\(b(X_t)\)&lt;/span&gt; is affine. The weight function &lt;span class=&quot;math&quot;&gt;\(\lambda(t)\)&lt;/span&gt; is typically chosen to be inverse proportional to &lt;span class=&quot;math&quot;&gt;\(\mathbb{E}[||\nabla_x \log p(X_t | X_0)||^2]\)&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The cool part of this technique is that the model is not trained on a single noise level, but multiple levels.&lt;/p&gt;
&lt;h3 id=&quot;124-solving-the-reverse-sde&quot;&gt;1.2.4 Solving the Reverse SDE&lt;/h3&gt;
&lt;p&gt;We can simulate the reverse process by solving the (estimate) reverse SDE with numerical SDE solvers, the simplest being the Euler-Maruyama Method. It involves discretizing the SDE using finite time steps and small Gaussian noise.&lt;/p&gt;
&lt;p&gt;Choose a small (negative) time step, initialize &lt;span class=&quot;math&quot;&gt;\(t \leftarrow T\)&lt;/span&gt; and iterate the following until &lt;span class=&quot;math&quot;&gt;\(t \approx 0\)&lt;/span&gt;&lt;/p&gt;
&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ΔX ← [b(X, t) - σ^2(t)s_θ(X, t)]Δt + σ(t)√|Δt|z_t
X ← X + ΔX
t ← t + Δt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;where, &lt;span class=&quot;math&quot;&gt;\(z_t \sim \mathcal{N}(0, I)\)&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id=&quot;2-training-a-time-dependent-score-based-model&quot;&gt;2 Training a Time-Dependent Score-Based Model&lt;/h2&gt;
&lt;h3 id=&quot;21-set-up&quot;&gt;2.1 Set-up&lt;/h3&gt;
&lt;p&gt;Let&#x27;s specify a SDE for demonstration&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[dX_t = \sigma_t dW, t \in [0, 1]\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;In this case,&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[p(X_t | X_0) = \mathcal{N}\left(X_0, \frac{1}{2\log \sigma} (\sigma^{2t} - 1)I\right)\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;When &lt;span class=&quot;math&quot;&gt;\(\sigma\)&lt;/span&gt; is large, at &lt;span class=&quot;math&quot;&gt;\(t = 1\)&lt;/span&gt;, the prior distribution, &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt; is&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[\int \pi(y)\mathcal{N}\left(y, \frac{1}{2\log \sigma} (\sigma^2 - 1)I\right)dy \approx \mathcal{N}\left(0, \frac{1}{2\log \sigma} (\sigma^2 - 1)I\right),\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;which is approximately independent of the data distribution and is easy to sample from.&lt;/p&gt;
&lt;p&gt;Intuitively, this SDE captures a continuum of Gaussian perturbations with zero mean and variance function &lt;span class=&quot;math&quot;&gt;\(\frac{1}{2\log \sigma} (\sigma^2 - 1)\)&lt;/span&gt;. This continuum of perturbations allows us to gradually transfer samples from a data distribution &lt;span class=&quot;math&quot;&gt;\(\pi\)&lt;/span&gt; to a simple Gaussian distribution &lt;span class=&quot;math&quot;&gt;\(p\)&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The reverse-time SDE for the above toy equation is given by&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[dX_t = -\sigma^{2t}\nabla \log p(X_t, t)dt + \sigma_t d\bar{W}.\]&lt;/span&gt;&lt;/p&gt;
&lt;h3 id=&quot;22-implementation-details&quot;&gt;2.2 Implementation Details&lt;/h3&gt;
&lt;p&gt;There are no restrictions on the network architecture of time-dependent score-based models, except that their output should have the same dimensionality as the input, and they should be conditioned on time. So,&lt;/p&gt;
&lt;!--
def loss_fn(model, x, marginal_prob_std, eps=1e-5):
    &quot;&quot;&quot;The loss function for training score-based generative models.

    Args:
      model: A PyTorch model instance that represents a
        time-dependent score-based model.
      x: A mini-batch of training data.
      marginal_prob_std: A function that gives the standard deviation of
        the perturbation kernel.
      eps: A tolerance value for numerical stability.
    &quot;&quot;&quot;
    random_t = torch.rand(x.shape[0], device=x.device) * (1.0 - eps) + eps
    z = torch.randn_like(x)
    std = marginal_prob_std(random_t)
    perturbed_x = x + z * std[:, None, None, None]
    score = model(perturbed_x, random_t)
    loss = torch.mean(
        torch.sum((score * std[:, None, None, None] + z) ** 2, dim=(1, 2, 3))
    )
    return loss


class GaussianFourierProjection(nn.Module):
    &quot;&quot;&quot;Gaussian random features for encoding time steps.&quot;&quot;&quot;

    def __init__(self, embed_dim, scale=30.0):
        super().__init__()
        # Randomly sample weights during initialization. These weights are fixed
        # during optimization and are not trainable.
        self.W = nn.Parameter(torch.randn(embed_dim // 2) * scale, requires_grad=False)

    def forward(self, x):
        x_proj = x[:, None] * self.W[None, :] * 2 * np.pi
        return torch.cat([torch.sin(x_proj), torch.cos(x_proj)], dim=-1)


class Dense(nn.Module):
    &quot;&quot;&quot;A fully connected layer that reshapes outputs to feature maps.&quot;&quot;&quot;

    def __init__(self, input_dim, output_dim):
        super().__init__()
        self.dense = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        return self.dense(x)[..., None, None]


class ScoreNet(nn.Module):
    &quot;&quot;&quot;A time-dependent score-based model built upon U-Net architecture.&quot;&quot;&quot;

    def __init__(self, marginal_prob_std, channels=[32, 64, 128, 256], embed_dim=256):
        &quot;&quot;&quot;Initialize a time-dependent score-based network.

        Args:
          marginal_prob_std: A function that takes time t and gives the standard
            deviation of the perturbation kernel p_{0t}(x(t) | x(0)).
          channels: The number of channels for feature maps of each resolution.
          embed_dim: The dimensionality of Gaussian random feature embeddings.
        &quot;&quot;&quot;
        super().__init__()
        # Gaussian random feature embedding layer for time
        self.embed = nn.Sequential(
            GaussianFourierProjection(embed_dim=embed_dim),
            nn.Linear(embed_dim, embed_dim),
        )
        # Encoding layers where the resolution decreases
        self.conv1 = nn.Conv2d(1, channels[0], 3, stride=1, bias=False)
        self.dense1 = Dense(embed_dim, channels[0])
        self.gnorm1 = nn.GroupNorm(4, num_channels=channels[0])
        self.conv2 = nn.Conv2d(channels[0], channels[1], 3, stride=2, bias=False)
        self.dense2 = Dense(embed_dim, channels[1])
        self.gnorm2 = nn.GroupNorm(32, num_channels=channels[1])
        self.conv3 = nn.Conv2d(channels[1], channels[2], 3, stride=2, bias=False)
        self.dense3 = Dense(embed_dim, channels[2])
        self.gnorm3 = nn.GroupNorm(32, num_channels=channels[2])
        self.conv4 = nn.Conv2d(channels[2], channels[3], 3, stride=2, bias=False)
        self.dense4 = Dense(embed_dim, channels[3])
        self.gnorm4 = nn.GroupNorm(32, num_channels=channels[3])

        # Decoding layers where the resolution increases
        self.tconv4 = nn.ConvTranspose2d(
            channels[3], channels[2], 3, stride=2, bias=False
        )
        self.dense5 = Dense(embed_dim, channels[2])
        self.tgnorm4 = nn.GroupNorm(32, num_channels=channels[2])
        self.tconv3 = nn.ConvTranspose2d(
            channels[2] + channels[2],
            channels[1],
            3,
            stride=2,
            bias=False,
            output_padding=1,
        )
        self.dense6 = Dense(embed_dim, channels[1])
        self.tgnorm3 = nn.GroupNorm(32, num_channels=channels[1])
        self.tconv2 = nn.ConvTranspose2d(
            channels[1] + channels[1],
            channels[0],
            3,
            stride=2,
            bias=False,
            output_padding=1,
        )
        self.dense7 = Dense(embed_dim, channels[0])
        self.tgnorm2 = nn.GroupNorm(32, num_channels=channels[0])
        self.tconv1 = nn.ConvTranspose2d(channels[0] + channels[0], 1, 3, stride=1)

        # The swish activation function
        self.act = lambda x: x * torch.sigmoid(x)
        self.marginal_prob_std = marginal_prob_std

    def forward(self, x, t):
        # Obtain the Gaussian random feature embedding for t
        embed = self.act(self.embed(t))
        # Encoding path
        h1 = self.conv1(x)
        ## Incorporate information from t
        h1 += self.dense1(embed)
        ## Group normalization
        h1 = self.gnorm1(h1)
        h1 = self.act(h1)
        h2 = self.conv2(h1)
        h2 += self.dense2(embed)
        h2 = self.gnorm2(h2)
        h2 = self.act(h2)
        h3 = self.conv3(h2)
        h3 += self.dense3(embed)
        h3 = self.gnorm3(h3)
        h3 = self.act(h3)
        h4 = self.conv4(h3)
        h4 += self.dense4(embed)
        h4 = self.gnorm4(h4)
        h4 = self.act(h4)

        # Decoding path
        h = self.tconv4(h4)
        ## Skip connection from the encoding path
        h += self.dense5(embed)
        h = self.tgnorm4(h)
        h = self.act(h)
        h = self.tconv3(torch.cat([h, h3], dim=1))
        h += self.dense6(embed)
        h = self.tgnorm3(h)
        h = self.act(h)
        h = self.tconv2(torch.cat([h, h2], dim=1))
        h += self.dense7(embed)
        h = self.tgnorm2(h)
        h = self.act(h)
        h = self.tconv1(torch.cat([h, h1], dim=1))

        # Normalize output
        h = h / self.marginal_prob_std(t)[:, None, None, None]
        return h



In summary: --&gt;

&lt;ul&gt;
&lt;li&gt;The U-net architecture is the most popular choice as the backbone of the score network &lt;span class=&quot;math&quot;&gt;\(s_\theta(x, t)\)&lt;/span&gt; and that is what is used for the toy example.&lt;/li&gt;
&lt;li&gt;Time information is incorporated via Gaussian random features. First, &lt;span class=&quot;math&quot;&gt;\(\omega \sim \mathcal{N}(0, s^2I)\)&lt;/span&gt; is sampled which is subsequently fixed for the model (i.e., not learnable). For a time step &lt;span class=&quot;math&quot;&gt;\(t\)&lt;/span&gt;, the corresponding Gaussian random feature is defined as&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[[\sin(2\pi\omega t); \cos(2\pi\omega t)],\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;where &lt;span class=&quot;math&quot;&gt;\([\vec{a}; \vec{b}]\)&lt;/span&gt; denotes the concatenation of vector &lt;span class=&quot;math&quot;&gt;\(\vec{a}\)&lt;/span&gt; and &lt;span class=&quot;math&quot;&gt;\(\vec{b}\)&lt;/span&gt;. This Gaussian random feature is used as an encoding for time step &lt;span class=&quot;math&quot;&gt;\(t\)&lt;/span&gt; so that the score network can condition on &lt;span class=&quot;math&quot;&gt;\(t\)&lt;/span&gt; by incorporating this encoding.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The output of the U-net is scaled by &lt;span class=&quot;math&quot;&gt;\(1/\sqrt{\mathbb{E}[||\nabla_X \log p(X_t | X_0)||^2]}\)&lt;/span&gt;. This is because the optimal &lt;span class=&quot;math&quot;&gt;\(s_\theta(X_t, t)\)&lt;/span&gt; has an &lt;span class=&quot;math&quot;&gt;\(\ell_2\)&lt;/span&gt;-norm close to &lt;span class=&quot;math&quot;&gt;\(\mathbb{E}[||\nabla_x \log p(X_t | X_0)]||^2\)&lt;/span&gt;, and the rescaling helps capture the norm of the true score.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;23-sampling-method&quot;&gt;2.3 Sampling Method&lt;/h3&gt;
&lt;p&gt;To sample from the trained time-dependent score-based model &lt;span class=&quot;math&quot;&gt;\(s_\theta(x, t)\)&lt;/span&gt;, a sample is first drawn from the prior distribution &lt;span class=&quot;math&quot;&gt;\(p \approx \mathcal{N}\left(x; 0, \frac{1}{2\log \sigma} (\sigma^2 - 1)I\right)\)&lt;/span&gt;. Then, the reverse-time SDE is solved using the Euler-Maruyama approach. When applied to the above reverse-time SDE, the following iteration rule is obtained&lt;/p&gt;
&lt;p&gt;&lt;span class=&quot;math display&quot;&gt;\[x_{t-\Delta t} = x_t + \sigma^{2t}s_\theta(x_t, t)\Delta t + \sigma_t\sqrt{\Delta t}z_t,\]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;where &lt;span class=&quot;math&quot;&gt;\(z_t \sim \mathcal{N}(0, I)\)&lt;/span&gt;.&lt;/p&gt;
&lt;!--
num_steps = 500

def Euler_Maruyama_Sampler(
    score_model,
    marginal_prob_std,
    diffusion_coeff,
    batch_size=64,
    num_steps=num_steps,
    device=&quot;cuda&quot;,
    eps=1e-3,
):
    &quot;&quot;&quot;Generate samples from score-based models with the Euler-Maruyama solver.

    Args:
      score_model: A PyTorch model that represents the time-dependent score-based model.
      marginal_prob_std: A function that gives the standard deviation of
        the perturbation kernel.
      diffusion_coeff: A function that gives the diffusion coefficient of the SDE.
      batch_size: The number of samplers to generate by calling this function once.
      num_steps: The number of sampling steps.
        Equivalent to the number of discretized time steps.
      device: &#x27;cuda&#x27; for running on GPUs, and &#x27;cpu&#x27; for running on CPUs.
      eps: The smallest time step for numerical stability.

    Returns:
      Samples.
    &quot;&quot;&quot;
    t = torch.ones(batch_size, device=device)
    init_x = (
        torch.randn(batch_size, 1, 28, 28, device=device)

        * marginal_prob_std(t)[:, None, None, None]
    )
    time_steps = torch.linspace(1.0, eps, num_steps, device=device)
    step_size = time_steps[0] - time_steps[1]
    x = init_x
    with torch.no_grad():
        for time_step in tqdm.notebook.tqdm(time_steps):
            batch_time_step = torch.ones(batch_size, device=device) * time_step
            g = diffusion_coeff(batch_time_step)
            mean_x = (
                x

                + (g**2)[:, None, None, None]
                * score_model(x, batch_time_step)
                * step_size
            )
            x = mean_x + torch.sqrt(step_size) * g[
                :, None, None, None
            ] * torch.randn_like(x)
    # Do not include any noise in the last sampling step.
    return mean_x

--&gt;

&lt;p&gt;Using 500 sampling steps and a decent amount of training, samples such as these can be generated on MNIST data,&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;/images/mnist.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;These are not perfect but a good baseline on how we can train and experiment with score-based generative modelling using SDEs.&lt;/p&gt;
&lt;h2 id=&quot;3-references&quot;&gt;3. References&lt;/h2&gt;
&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class=&quot;nc&quot;&gt;@misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;song2021scorebasedgenerativemodelingstochastic&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;title&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{Score-Based Generative Modeling through Stochastic Differential Equations}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;author&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{Yang Song and Jascha Sohl-Dickstein and Diederik P. Kingma and Abhishek Kumar and Stefano Ermon and Ben Poole}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;year&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{2021}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;eprint&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{2011.13456}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;archivePrefix&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{arXiv}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;primaryClass&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{cs.LG}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;w&quot;&gt;      &lt;/span&gt;&lt;span class=&quot;na&quot;&gt;url&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;{https://arxiv.org/abs/2011.13456}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://yang-song.net/blog/2021/score/&quot;&gt;Yang Song&#x27;s Blog&lt;/a&gt; &quot;delves&quot; into this in excruciating detail and there are more example notebooks you can try.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;div class=&quot;footnote&quot;&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id=&quot;fn:1&quot;&gt;
&lt;p&gt;this is actually an abuse of notation&amp;#160;&lt;a class=&quot;footnote-backref&quot; href=&quot;#fnref:1&quot; title=&quot;Jump back to footnote 1 in the text&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description>
  </item>
</channel>
</rss>
