<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

 <title>Lisa Z.</title>
 <link href="https://chenyi00.github.io/atom.xml" rel="self"/>
 <link href="https://chenyi00.github.io/"/>
 <updated>2021-06-04T00:18:02+00:00</updated>
 <id>https://chenyi00.github.io</id>
 <author>
   <name>Lisa Zhu</name>
   <email></email>
 </author>

 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.11</title>
   <link href="https://chenyi00.github.io/2021/04/11/final-experiment/"/>
   <updated>2021-04-11T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2021/04/11/final-experiment</id>
   <content type="html">&lt;h3 id=&quot;bpe&quot;&gt;BPE&lt;/h3&gt;

&lt;p&gt;Based on previous outputs, there seemed to be a problem where there are a lot of unknown tokens in the output translation (see the last line of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;synthesis/train_model/pass-1/fs_generate*.out&lt;/code&gt;, where &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BLEU = 1.87&lt;/code&gt; for example). It was then recommended that the corpus all be tokenized before training. The results did improve significantly, and the third pass (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;synthesis/train_model/pass-3&lt;/code&gt;), I got &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BLEU = 10.92&lt;/code&gt;. Still not great, but at least greatly improved.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>I'm Back!</title>
   <link href="https://chenyi00.github.io/2021/03/20/hiatus/"/>
   <updated>2021-03-20T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2021/03/20/hiatus</id>
   <content type="html">&lt;p&gt;So it’s been a bit since I made my last post… The past two months have been kind of insane with classes, schoolwork, and perhaps most importantly, the much dreaded summer internship search 👻. But I finally got an offer, and I am now ready to get back on track with blogging!&lt;/p&gt;

&lt;!--more--&gt;

&lt;h3 id=&quot;hey-look-ma-i-made-it&quot;&gt;Hey Look Ma, I Made It&lt;/h3&gt;

&lt;p&gt;Like I said before, it has been a very difficult semester with my summer internship search. Lost count of how many times I got ghosted and/or rejected, and I was getting ready to just do research for the coming summer. Perhaps the internship gods decided that I was going to be the lucky cat of the year, and a recruiter from LinkedIn reached out with an opportunity, and somehow I made it through the résumé screen, the OA, the recruiter call, and the final three rounds of virtual onsite interviews… I cannot believe it. Even as I sit here looking at the offer, I am still in shock. Guess I graduated from the kiddie school and am now about to take the first step from the bubble into the real world.&lt;/p&gt;

&lt;p&gt;All else aside, this has given me hope. There are so many new things to learn and I am beyond excited and grateful. Maybe I will move to New York City for the summer. Maybe I will get to meet some fellow interns/colleagues. Maybe I will start some serious photography projects. So many possibilities, so little time…&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.10</title>
   <link href="https://chenyi00.github.io/2021/01/25/get-synthetic-corpus/"/>
   <updated>2021-01-25T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2021/01/25/get-synthetic-corpus</id>
   <content type="html">&lt;h3 id=&quot;generating-synthetic-corpus&quot;&gt;Generating Synthetic Corpus&lt;/h3&gt;
&lt;p&gt;Following from the last post, it was discussed that no alignment situations are inevitable due to noisy data and such. I therefore edited the script to skip the words that are not aligned. In addition, I ran the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;moses&lt;/code&gt; tokenizer (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;tokenize_parallel.sh&lt;/code&gt;) on the parallel corpus as well just in case (even though the website said it is already tokenized). While I was successful in generating synthetic corpus after this modification, it was strangely repetitive. Note that for each terminology, we keep a list of words from the parallel corpus that it can replace. However, it looks like the script loops over a specific word a number of times instead of all of the words.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h3 id=&quot;finding-bugs&quot;&gt;Finding Bugs&lt;/h3&gt;
&lt;p&gt;The issue occurs in the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;generate_new_sentence_pairs&lt;/code&gt; function, where the index is not updating properly, causing the word at index 0 to be returned for every iteration of the loop. However, there are other concerns. Since the goal is to generate a new parallel corpus with those terms included, it is necessary to overhaul the entire function so that it does not only spit out individual sentences. I am currently still working on this new function.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.9</title>
   <link href="https://chenyi00.github.io/2021/01/11/wikipedia-rerun-moses/"/>
   <updated>2021-01-11T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2021/01/11/wikipedia-rerun-moses</id>
   <content type="html">&lt;h3 id=&quot;using-moses&quot;&gt;Using &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;moses&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;The monolingual datasets and terminologies have been tokenized using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;moses&lt;/code&gt; tokenizer and the special characters from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;HTML&lt;/code&gt; have been replaced. A new experiment is now being run with the pre-tokenized datasets.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h3 id=&quot;results&quot;&gt;Results&lt;/h3&gt;
&lt;p&gt;Now there are embeddings generated for terms such as &lt;strong&gt;covid19&lt;/strong&gt; and &lt;strong&gt;H1N1&lt;/strong&gt;. The new tokenizer seems to work well. There are still some terms that do not have embeddings generated:&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;&lt;strong&gt;FRENCH&lt;/strong&gt;&lt;/th&gt;
      &lt;th&gt;&lt;strong&gt;ENGLISH&lt;/strong&gt;&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;c19 (472)&lt;/td&gt;
      &lt;td&gt;c19 (5199)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;cv19 (11)&lt;/td&gt;
      &lt;td&gt;cv19 (34)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;cov19 (4)&lt;/td&gt;
      &lt;td&gt;cov19 (20)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;CV-19 (0)&lt;/td&gt;
      &lt;td&gt;CV-19 (0)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Isopropanol (62)&lt;/td&gt;
      &lt;td&gt;Isopropanol (208)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;vivid-19 (0)&lt;/td&gt;
      &lt;td&gt;vivid-19 (0)&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;Télétravail (311)&lt;/td&gt;
      &lt;td&gt;WFH (270)&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;There are also three pairs of terms: &lt;strong&gt;corona/corona&lt;/strong&gt;, &lt;strong&gt;germicide/germicide&lt;/strong&gt;, and &lt;strong&gt;Wuhan/Wuhan&lt;/strong&gt; whose similar terms could not be found in the parallel corpora provided by &lt;strong&gt;EMEA&lt;/strong&gt;, and hence do not have replacements.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.8</title>
   <link href="https://chenyi00.github.io/2021/01/04/debugging-tokenizer/"/>
   <updated>2021-01-04T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2021/01/04/debugging-tokenizer</id>
   <content type="html">&lt;h3 id=&quot;follow-up-on-embedding-anomaly&quot;&gt;Follow-up on Embedding Anomaly&lt;/h3&gt;
&lt;p&gt;As mentioned in the previous post, there seems to be an issue when generating embeddings even if the term occured many times in both monolingual corpora. To investigate further, I plan to run the script on the term &lt;strong&gt;covid19&lt;/strong&gt; only.&lt;/p&gt;

&lt;!--more--&gt;

&lt;h3 id=&quot;eureka&quot;&gt;Eureka&lt;/h3&gt;
&lt;p&gt;The results still indicated no embeddings were generated for &lt;strong&gt;covid19&lt;/strong&gt;. After a meeting with Huda, we figured out that it was the tokenizer that was having issues. The one originally used was &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;gensim.utils.tokenize()&lt;/code&gt;, which removes numbers - not hard to see why that would cause a problem.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.7</title>
   <link href="https://chenyi00.github.io/2020/12/28/frequency-count/"/>
   <updated>2020-12-28T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/12/28/frequency-count</id>
   <content type="html">&lt;h3 id=&quot;running-with-wikipedia-datasets&quot;&gt;Running with Wikipedia Datasets&lt;/h3&gt;
&lt;p&gt;There were some issues with running with the wikipedia datasets as the output was still a bit unsatisfactory. For example, no embeddings were generated for critical terms such as &lt;strong&gt;COVID&lt;/strong&gt; and &lt;strong&gt;cov19&lt;/strong&gt; (along with other variants). The Wikipedia dumps, however, should have contained these terms because of how recent they were (11/20/2020). As suggested by advisors, it would be a good idea to count the frequency of these terms in the monoloingual corpora.
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;counting-frequency&quot;&gt;Counting Frequency&lt;/h3&gt;
&lt;p&gt;With the Facebook dataset in English and French and a Python script, I counted the occurences of each term (single and multi-word ones) in the respective Wikipedia datasets. 27 (0.75%) terms in the English dataset and 45 (12.5%) in the French dataset never appeared in their respective monolingual corpora. Curiously, while there was embedding generated for the pair &lt;strong&gt;pandemic (50843)/pandémie (9326)&lt;/strong&gt;, there was none for the pair &lt;strong&gt;covid19 (55440)/covid19 (10175)&lt;/strong&gt;, both of which occurred more frequently than the former.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.6</title>
   <link href="https://chenyi00.github.io/2020/11/29/wikipedia-dump/"/>
   <updated>2020-11-29T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/11/29/wikipedia-dump</id>
   <content type="html">&lt;h3 id=&quot;wikipedia-dumps&quot;&gt;Wikipedia Dumps&lt;/h3&gt;
&lt;p&gt;Of course, Wikipedia always has some of the most up-to-date information on any major world events, including the current pandemic crisis. I have been hesitant to use it because of how the dumps are structured - they are in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;XML&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SQL&lt;/code&gt; formats, and I did not know how to extract information from them and save it in text file format. Luckily, Matt showed me this helpful &lt;a href=&quot;https://github.com/tatuylonen/wiktextract&quot;&gt;repo&lt;/a&gt; that builds a tool for parsing these dumps. However, the documentation has said that this script is for parsing Wikitionaries and for English only, and I needed one for general articles in different languages. I looked around and found the &lt;a href=&quot;https://wiki.apertium.org/wiki/Wikipedia_Extractor&quot;&gt;&lt;strong&gt;Wikipedia Extractor&lt;/strong&gt;&lt;/a&gt; script which should do just that. This gave me some hope - I downloaded the gigantic &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.bz2&lt;/code&gt; dumps for &lt;a href=&quot;https://dumps.wikimedia.org/enwiki/20201120/&quot;&gt;English&lt;/a&gt; and &lt;a href=&quot;https://dumps.wikimedia.org/frwiki/20201120/&quot;&gt;French&lt;/a&gt; and ran the script to generate plain text files and remove formatting.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.5</title>
   <link href="https://chenyi00.github.io/2020/11/22/commoncrawl-rerun/"/>
   <updated>2020-11-22T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/11/22/commoncrawl-rerun</id>
   <content type="html">&lt;h3 id=&quot;using-other-datasets&quot;&gt;Using Other Datasets&lt;/h3&gt;
&lt;p&gt;I soon ran into the problem where the CommonCrawl datasets available to use were a bit too messy &amp;amp; not yet processed, and the clean data were not recent enough to contain the terminologies. While the hunt for good datasets to use is still on, I tried NeuLab’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;covid-datashare&lt;/code&gt; &lt;a href=&quot;https://github.com/neulab/covid19-datashare&quot;&gt;repository&lt;/a&gt;. The datasets here were not quite long enough, and did not contain all the single-word terminologies as I had hoped, causing the program to crash because certain embeddings could not be generated.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.4</title>
   <link href="https://chenyi00.github.io/2020/10/26/moving-forward/"/>
   <updated>2020-10-26T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/10/26/moving-forward</id>
   <content type="html">&lt;h3 id=&quot;tips-for-moving-forward&quot;&gt;Tips for moving forward&lt;/h3&gt;
&lt;p&gt;Things to keep in mind after meeting with the mentors today:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The current script only handles single-words, and for now keep it this way
    &lt;ul&gt;
      &lt;li&gt;Filter out multi-word terminologies and only keep the single-word ones&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;For the monolingual data input to the script, look to &lt;strong&gt;CommonCrawl&lt;/strong&gt; (more abundant) or &lt;strong&gt;Wikipedia&lt;/strong&gt; (need to choose the more recent datasets for including COVID-19 terminologies)&lt;/li&gt;
  &lt;li&gt;For future references - possible ways to find replacement for phrases:
    &lt;ul&gt;
      &lt;li&gt;Basic approach: average the embeddings of terminologies and attempt to replace single words with these phrases in the parallel corpus, logic is reasonable enough&lt;/li&gt;
      &lt;li&gt;More sophisticated approach: align the terminologies with the corpus and somehow extract phrases to replace; this will be a lot more involved and complicated
&lt;!--more--&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;points-of-confusion&quot;&gt;Points of confusion&lt;/h3&gt;
&lt;p&gt;I had a misconception regarding how embeddings are generated. The embeddings are generated for the terminologies, not the parallel corpus, but via the monolingual corpus input. We find similar pairs to the terminologies in the monolingual data embeddings, and if these similar pairs exist in the parallel data, compute similarity between the similar pairs and the original glossary terms.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.3</title>
   <link href="https://chenyi00.github.io/2020/10/24/benchmark-questions/"/>
   <updated>2020-10-24T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/10/24/benchmark-questions</id>
   <content type="html">&lt;h3 id=&quot;questions-that-remain&quot;&gt;Questions that remain&lt;/h3&gt;
&lt;p&gt;From the benchmark:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;What could be the cause of the low similarity level in generated replacement pairs?
    &lt;ul&gt;
      &lt;li&gt;I used the EMEA dataset as both parallel corpus and monolingual corpora inputs to the program, yet they may not contain all the glossary terms and their translations, could that be the problem?&lt;/li&gt;
      &lt;li&gt;Question for Shuhao: do the TICO-19 datasets contain all the glossary terms? Perhaps that would be a more appropriate monolingual corpora set to use?&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;How to get the model to work with multi-word glossary terms?
    &lt;ul&gt;
      &lt;li&gt;The current generated embeddings from training on corpus seem to only work for single words, perhaps modify to account for multiple words?&lt;/li&gt;
      &lt;li&gt;Match the glossary terms (with the ECO method) to newly generated embeddings?&lt;/li&gt;
      &lt;li&gt;Other approaches?
&lt;!--more--&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;From the paper:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;How are the edge cases handled when generating embeddings? For fixed window size $c$, how are the embeddings for - let’s say, the first and last word of the phrase - computed?&lt;/li&gt;
  &lt;li&gt;How to integrate this into the current program? Perhaps generate single embeddings for the terminologies and find a match? Still not sure exactly how to handle this…&lt;/li&gt;
&lt;/ul&gt;
</content>
 </entry>
 
 <entry>
   <title>Literature Review Notes | ECO</title>
   <link href="https://chenyi00.github.io/2020/10/23/eco-embedding/"/>
   <updated>2020-10-23T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/10/23/eco-embedding</id>
   <content type="html">&lt;p&gt;As mentioned in the previous post, I did not know how to generate embeddings for $n$-gram glossary terms. This paper can hopefully shed light on how to approach this problem.&lt;/p&gt;

&lt;h3 id=&quot;efficient-compositional-order-sensitive-n-gram-embeddings&quot;&gt;Efficient, Compositional, Order-Sensitive $n$-gram Embeddings&lt;/h3&gt;
&lt;p&gt;The ECO method creates decompositional embeddings for words offline and combines them to create new embeddings for phrases in real time. Upside: ECO can create embeddings for phrases not seen during training.
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;word2vec-model&quot;&gt;Word2Vec model&lt;/h3&gt;
&lt;p&gt;Word2Vec consists of four possible models: &lt;strong&gt;CBOW&lt;/strong&gt; (Continuous Bag-of-Words) with hierarchical softmax or negative sampling, and &lt;strong&gt;SG&lt;/strong&gt; (Skip-Gram) with the same choices. The former predicts a single word $w$ surrounded by the given context, the latter context words given $w$. Given a token $s_j$, we calculate the log-probability (softmax over inner products of the embeddings of the two tokens) of the context words $s_k$, $k \in [j - c, 0) \cup (0, j + c]$ for a window size $c$ (half of context size) and average across all $s_j \in s$, and finally average again across all $s \in S$, which gives us:&lt;/p&gt;

\[\frac{1}{|S|}\sum_{s \in S}\frac{1}{|s|}\sum_j\sum_k\log(p(s_k|s_j))\]

&lt;p&gt;with probability distribution:&lt;/p&gt;

\[p(s_k | s_j) = \frac{\exp(\langle v_{s_k}^{out}, v_{s_j}\rangle)}{\exp(\sum_{w \in W}\langle v_w^{out}, v_{s_j} \rangle)}\]

&lt;p&gt;where $v$ and $v_{out}$ denote input and output representations of a word. The concept of input v.s. output confused me, but this &lt;a href=&quot;https://tinyurl.com/y343td7d&quot;&gt;StackExchange&lt;/a&gt; thread cleared it up a bit. First we map the input one-hot vectors to word embeddings, and then map the hidden layer to the output vector of vocabulary size.&lt;/p&gt;

&lt;h3 id=&quot;rudimentary-approaches&quot;&gt;Rudimentary approaches&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;Treat an $n$-gram as a single word: just preprocess text and run Word2Vec accordingly; cannot embed unkown $n$-grams even if each of the words actually appeared in the raining corpus&lt;/li&gt;
  &lt;li&gt;Simple average over embeddings: compute embedding for each word and average them; does not take into account the ordering of words in the $n$-gram&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;eco&quot;&gt;ECO&lt;/h3&gt;
&lt;p&gt;Based off of SG, we are going to define &lt;strong&gt;SE&lt;/strong&gt; (Skip-Embeddings) that will split up the context around a word into multiple categories. Parametrize each word $w$ with $2c$ embeddings each with their own objective function:&lt;/p&gt;

\[\frac{1}{|S|}\sum_{s \in S}\frac{1}{|s|}\sum_j\log(p(s_k|s_j))\]

&lt;p&gt;where $s_k$ is the word $i$ positions away from $s_j$ in $s$, $i \in [-c, 0) \cup (0, c]$, and a negative $i$ means to the left, positive to the right of $w$. The new probability distribution is:&lt;/p&gt;

\[p(s_k | s_j) = \frac{\exp(\langle v_{s_k}^{i_{out}}, v_{s_j}^i\rangle)}{\exp(\sum_{w \in W}\langle v_w^{i_{out}}, v_{s_j}^i \rangle)}\]

&lt;p&gt;All $2c$ of these decompositional embeddings created per word are SEs. Since a skip-embedding only accounts for a single token of distance $i$ from $w$, the dimensionality of $v_w^i$ should be kept to $d/2c$ (as opposed to $d$ as used by Word2Vec). Each SE is trained with only $d/2c$ parameters.&lt;/p&gt;

&lt;p&gt;To actually embed the $n$-gram after creating these SEs, we average the SEs and divide them into two vectors: $v^{L}$ and $v^{R}$ that summarize left and right contexts of the $n$-gram independently, computed as follows:&lt;/p&gt;

\[v^{L}_{[w_1, w_n]} = \frac{v_{w_1}^{-n} + \ldots + v_{w_n}^{-1}}{n}\]

\[v^{R}_{[w_1, w_n]} = \frac{v_{w_1}^{1} + \ldots + v_{w_n}^{n}}{n}\]

&lt;p&gt;Concatenate the two vectors to create a single embedding of the entire $n$-gram. Note that after concatenation we have a dimentionality of $d/c$.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.2</title>
   <link href="https://chenyi00.github.io/2020/10/22/benchmark-failure/"/>
   <updated>2020-10-22T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/10/22/benchmark-failure</id>
   <content type="html">&lt;h3 id=&quot;modifying-script&quot;&gt;Modifying script&lt;/h3&gt;
&lt;p&gt;I started with creating a new script &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;benchmark.py&lt;/code&gt; (based on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;synthesis.py&lt;/code&gt;), which can handle two parallel files with the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;--monolingual-glossary&lt;/code&gt; flag - provided that they are aligned line by line (in this case, specifically for the Facebook datasets). 
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;initial-run-failure&quot;&gt;Initial run failure&lt;/h3&gt;
&lt;p&gt;After the initial training of the script, I remembered that the current script only works on glossaries with single words, which is the reason why there were few embeddings generated. For example, here are a few lines in the output:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;Grippe de 1918 1918 flu 	 no embeddings
bronchite aiguë acute bronchitis 	 no embeddings&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;As you can see, in the original English term we have bigrams. No word embeddings are generated for either pair. There are some rare instances of embeddings that do get generated, but not only are they just unigrams, they also have abysmal similarity ratings (calculated as: $sim(f_{gloss}, f_{match}) \cdot sim(e_{gloss}, e_{match})$).&lt;/p&gt;
&lt;details&gt;
&lt;summary&gt;Click to see sample output&lt;/summary&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;SIDA AIDS 	 pruned 	 1280 	 337 	 &lt;span class=&quot;o&quot;&gt;[{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'maladie'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'disease'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.10634502472876672, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 20&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'familiaux'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'family'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.09439994727364365, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 27&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'sévère'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'severe'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.06758738560884492, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 22&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'antécédents'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'history'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.061721948986114494, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 18&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'patients'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'patients'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.05183887041854973, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 250&lt;span class=&quot;o&quot;&gt;}]&lt;/span&gt;

aérien airway 	 pruned 	 5120 	 224 	 &lt;span class=&quot;o&quot;&gt;[{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'trouble'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'disorder'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.06756156498017774, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 18&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'symptômes'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'symptoms'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.055993081384110965, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 97&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'raideur'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'stiffness'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.05032308793326834, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 27&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'psychotiques'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'psychotic'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.04466918184504509, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 19&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'cérébrovasculaires'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'cerebrovascular'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.04236828493397127, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 36&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;, &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'f'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'cardiovasculaire'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'e'&lt;/span&gt;: &lt;span class=&quot;s1&quot;&gt;'cardiovascular'&lt;/span&gt;, &lt;span class=&quot;s1&quot;&gt;'similarity'&lt;/span&gt;: 0.040491460778032895, &lt;span class=&quot;s1&quot;&gt;'count'&lt;/span&gt;: 27&lt;span class=&quot;o&quot;&gt;}]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;


Perhaps these ratings are meant to be bad - after all we are looking at a glossary list with terms never seen before, even in a medicine-related parallel corpus with a high-resource language pair.
&lt;/details&gt;

&lt;h3 id=&quot;todo&quot;&gt;TODO&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;Figure out a way to create embeddings for multi-word phrases:
    &lt;ul&gt;
      &lt;li&gt;Even when phrases are of different lengths ($n \in [1, 7]$ for an $n$-gram in the TICO-19 glossary by Facebook), if we choose the max $n$ we should still be able to match shorter phrases&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;https://www.cs.jhu.edu/~huda/&quot;&gt;Huda&lt;/a&gt;’s suggestion: average the embeddings
        &lt;blockquote&gt;
          &lt;p&gt;this work from a few years ago looks at &lt;a href=&quot;https://azpoliak.github.io/publications/eco--eacl.pdf&quot;&gt;phrase embeddings&lt;/a&gt; but there is probably something newer, and I would start with just averaging&lt;/p&gt;
        &lt;/blockquote&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Read the suggested paper on &lt;a href=&quot;https://azpoliak.github.io/publications/eco--eacl.pdf&quot;&gt;phrase embeddings&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content>
 </entry>
 
 <entry>
   <title>Synthesis Experiments | Benchmark Pt.1</title>
   <link href="https://chenyi00.github.io/2020/10/21/choosing-dataset/"/>
   <updated>2020-10-21T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/10/21/choosing-dataset</id>
   <content type="html">&lt;h3 id=&quot;selecting-training-datasets&quot;&gt;Selecting training datasets&lt;/h3&gt;
&lt;p&gt;Considering the fact that we will mainly be working with medical-related terminologies, I went on &lt;a href=&quot;http://opus.nlpl.eu/&quot;&gt;&lt;strong&gt;OPUS&lt;/strong&gt;&lt;/a&gt; to search for datasets that are more appropriate for this purpose than the EN-DE toy dataset. I came across &lt;a href=&quot;http://opus.nlpl.eu/EMEA.php&quot;&gt;&lt;strong&gt;EMEA&lt;/strong&gt;&lt;/a&gt; - “a parallel corpus made out of PDF documents from the European Medicines Agency”. I chose the EN-FR datasets (in nice &lt;strong&gt;MOSES&lt;/strong&gt; format) to experiment with, partly because I am familiar with both languages.
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;todo&quot;&gt;TODO&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;Modify &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;synthesis.py&lt;/code&gt; to take in two files for terminologies provided by Facebook&lt;/li&gt;
  &lt;li&gt;Run experiment to get preliminary results&lt;/li&gt;
&lt;/ul&gt;
</content>
 </entry>
 
 <entry>
   <title>Literature Review Notes | TICO-19</title>
   <link href="https://chenyi00.github.io/2020/09/20/tico-set/"/>
   <updated>2020-09-20T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/09/20/tico-set</id>
   <content type="html">&lt;h3 id=&quot;translation-initiative-for-covid-19&quot;&gt;Translation Initiative for COvid-19&lt;/h3&gt;

&lt;p&gt;With a focus on low-resource languages, TICO-19 aims to make available data in 35 different lanuages (of which 26 lesser resourced languages in addition to the 9 “pivot” languages) for researchers and to assist the development of tools to effectively disseminate critical information for disaster situations such as presented by the current global pandemic. 
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;dataset-development-methodology&quot;&gt;Dataset development methodology&lt;/h3&gt;

&lt;p&gt;TICO-19 was created with the following benchmark:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Sampling from a variety of public sources - such as &lt;strong&gt;PubMed&lt;/strong&gt;, &lt;strong&gt;Wikipedia&lt;/strong&gt; (and related applications), &lt;strong&gt;CMU English-Haitian Creole dataset&lt;/strong&gt;, etc. - covering information domains including but not restricted to: news, testing, travel advisories, scientific development.&lt;/li&gt;
  &lt;li&gt;Grouping languages into three main categories:
    &lt;ul&gt;
      &lt;li&gt;&lt;strong&gt;Pivots&lt;/strong&gt;: resource-rich languages which sometimes serve as an intermediary between English and a target language&lt;/li&gt;
      &lt;li&gt;&lt;strong&gt;Priority&lt;/strong&gt;: 18 languages classified as high-priority by the &lt;strong&gt;TWB&lt;/strong&gt; due to the overwhelming requests they are receiving &amp;amp; “the strategic location of their partners (e.g. the Red Cross)”&lt;/li&gt;
      &lt;li&gt;&lt;strong&gt;Important&lt;/strong&gt;: 8 languages spoken (by millions) in South and South-East Asia&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Ensuring the quality of translation through a two-step human quality control process:
    &lt;ul&gt;
      &lt;li&gt;Each document is translated by lanaguage service providers (&lt;strong&gt;LSP&lt;/strong&gt;) and edited&lt;/li&gt;
      &lt;li&gt;A selected fraction of the data (and all content from &lt;strong&gt;PubMed&lt;/strong&gt; since it is the hardest to translate) goes through another round of inspection&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;developer-resources&quot;&gt;Developer resources&lt;/h3&gt;

&lt;p&gt;Additional COVID-19 data are made available in forms of monolingual &amp;amp; parallel corpus gathered from Wikipedia, BBC, VOA, and NGOs, national/state government sources, respectively.&lt;/p&gt;

&lt;h3 id=&quot;discussion--future-work&quot;&gt;Discussion &amp;amp; future work&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Aside from the baseline results, note that there is a lack of pre-trained MT systems on the low-source languages such as Dari, Myanmar, Oromo, isiZulu, etc. This, along with the difference in &lt;strong&gt;BLEU&lt;/strong&gt; scores between pivot &amp;amp; non-pivot languages highlights the dire need to collect more data for under-represented communities and languages.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;While current MT systems are mostly trained on general-purpose data or particular out-of-domain data, domain adaptation techniques should improve performance. Incorporating terminologies previously unseen in the vocabulary could also help ensure the quality of translation.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Multilingual NMT systems trained on massive corpus from the web are also promising in improving translation quality for languages in the lower end of data availability.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Further efforts to increase representation of low-resource lanagues in the public domain corpora in order to prepare for future crises where translation technologies will be needed.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read more about the TICO-19 effort &lt;a href=&quot;https://tico-19.github.io/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>CLSP Cluster Notes | Day 6</title>
   <link href="https://chenyi00.github.io/2020/09/03/fairseq-generate/"/>
   <updated>2020-09-03T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/09/03/fairseq-generate</id>
   <content type="html">&lt;h3 id=&quot;fairseq-generate&quot;&gt;fairseq-generate&lt;/h3&gt;

&lt;p&gt;Other than the fact that I did not request a GPU in the beginning (which I did not think it needed), the script ran smoothly - only took me two tries of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;qsub&lt;/code&gt;! Here are some outputs (DE -&amp;gt; EN):&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;&lt;strong&gt;GERMAN&lt;/strong&gt;&lt;/th&gt;
      &lt;th&gt;&lt;strong&gt;ENGLISH&lt;/strong&gt;&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;darf ich ehrlich sein ?&lt;/td&gt;
      &lt;td&gt;can i be honest ?&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;mache einfach deinen job .&lt;/td&gt;
      &lt;td&gt;just do your job .&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;die ganze umwelt@@ diskussion .&lt;/td&gt;
      &lt;td&gt;the whole environmental discussion .&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;oh mein gott .&lt;/td&gt;
      &lt;td&gt;oh , my god .&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;die sonne geht auf&lt;/td&gt;
      &lt;td&gt;the sun is ris@@ ing .&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;:)&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>CLSP Cluster Notes | Day 5</title>
   <link href="https://chenyi00.github.io/2020/09/02/fairseq-train/"/>
   <updated>2020-09-02T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/09/02/fairseq-train</id>
   <content type="html">&lt;h3 id=&quot;fairseq-train&quot;&gt;fairseq-train&lt;/h3&gt;

&lt;p&gt;To train the model, in addition to the tutorial on fairseq’s webpage, I had to specify an optimizer. I went with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SGD&lt;/code&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the computational burden, achieving &lt;strong&gt;faster&lt;/strong&gt; iterations in trade for a &lt;strong&gt;lower&lt;/strong&gt; convergence rate.
&lt;!--more--&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Probably should have chosen &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Adam&lt;/code&gt;, which seems more popular now. Perhaps I will train the model on Adam as well and compare output.&lt;/p&gt;

&lt;p&gt;I set &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;--max-tokens&lt;/code&gt; to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;500&lt;/code&gt; - not sure if &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;4000&lt;/code&gt; would be too much to ask of the GPU.&lt;/p&gt;

&lt;p&gt;Speaking of GPU, I also included Guanghui’s script to acquire one.&lt;/p&gt;

&lt;p&gt;Job stopped on the 18th epoch. I will go do my homework now - &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq-generate&lt;/code&gt; coming up next.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>CLSP Cluster Notes | Day 4</title>
   <link href="https://chenyi00.github.io/2020/08/31/fairseq-preprocess/"/>
   <updated>2020-08-31T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/08/31/fairseq-preprocess</id>
   <content type="html">&lt;div class=&quot;message&quot;&gt;
    The fall semester started today, hopefully I can keep up with learning how to work with the grid!
&lt;/div&gt;

&lt;p&gt;I struggled quite a bit with the fairseq commands today (and job scheduling). I realized that I could not run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq-interactive&lt;/code&gt; with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;qlogin&lt;/code&gt; since the latter was not granting me a session, and the former was too heavy a job. So, in the end I decided to non-interactively train a model instead.
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;fairseq-preprocess&quot;&gt;fairseq-preprocess&lt;/h3&gt;

&lt;p&gt;I submitted a lot of jobs, but most of them failed to execute thoroughly. Most of it had to do with directories - specifying &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;-wd&lt;/code&gt; proved to be an annoyance, and the place I wanted the output to be saved ended up being where some bash scripts were saved after execution as well, so the program had trouble looking for the necessary files it needed. In the end I gave up and set the working directory to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq/examples/translation&lt;/code&gt; and it worked out.&lt;/p&gt;

&lt;p&gt;Next steps include actually training the model (and figuring out how to acquire GPUs) and then generating output.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>CLSP Cluster Notes | Day 3</title>
   <link href="https://chenyi00.github.io/2020/08/30/fairseq-install/"/>
   <updated>2020-08-30T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/08/30/fairseq-install</id>
   <content type="html">&lt;h3 id=&quot;installing-fairseq&quot;&gt;Installing fairseq&lt;/h3&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/pytorch/fairseq&quot;&gt;&lt;strong&gt;Fairseq&lt;/strong&gt;&lt;/a&gt; is a Python-based language modeling toolkit devloped by Facebook (I was also recommended &lt;a href=&quot;https://marian-nmt.github.io/&quot;&gt;&lt;strong&gt;Marian&lt;/strong&gt;&lt;/a&gt;, which is C++-based and developed by Microsoft, but fairseq was easier to install, and Python is all the rage these days).&lt;/p&gt;

&lt;p&gt;For the record, the CLSP grid runs Python2 instead of Python3 as default, so I inevitably came across the problem at the last step:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;git clone https://github.com/pytorch/fairseq
&lt;span class=&quot;nb&quot;&gt;cd &lt;/span&gt;fairseq
pip &lt;span class=&quot;nb&quot;&gt;install&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--editable&lt;/span&gt; ./&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Just reminding myself to use &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pip3&lt;/code&gt; instead of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pip&lt;/code&gt; next time.
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;fairseq-command-line-tools&quot;&gt;Fairseq command-line tools&lt;/h3&gt;

&lt;p&gt;Ran into a question about the difference between &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq-generate&lt;/code&gt; &amp;amp; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq-interactive&lt;/code&gt;, and found this:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq-generate&lt;/code&gt; is for batched data and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fairseq-interactive&lt;/code&gt; is for streaming data (and interactive use). Otherwise they are very similar.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In this case, it would make more sense to use the former when writing scripts &amp;amp; scheduling jobs.&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>CLSP Cluster Notes | Day 2</title>
   <link href="https://chenyi00.github.io/2020/08/29/conda-install/"/>
   <updated>2020-08-29T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/08/29/conda-install</id>
   <content type="html">&lt;h3 id=&quot;installing-miniconda3&quot;&gt;Installing miniconda3&lt;/h3&gt;

&lt;p&gt;Used &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;wget&lt;/code&gt; instead of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;curl&lt;/code&gt; - the latter initially gave me the impression that the program went into some infinite loop with the following message:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Please answer ‘yes’ or ‘no’:&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The previous ran fine, although I thought it froze when I executed the shell script. Turned out I just needed to be patient and wait…
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;activating-conda-environment&quot;&gt;Activating conda environment&lt;/h3&gt;

&lt;p&gt;From the install message:&lt;/p&gt;
&lt;blockquote&gt;
  &lt;p&gt;To activate conda’s base environment in your current shell session:&lt;/p&gt;
&lt;/blockquote&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;&lt;span class=&quot;nb&quot;&gt;eval&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;$(&lt;/span&gt;/PATH_TO_CONDA/miniconda3/bin/conda shell.YOUR_SHELL_NAME hook&lt;span class=&quot;si&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;blockquote&gt;
  &lt;p&gt;To install conda’s shell functions for easier access, first activate, then:&lt;/p&gt;
&lt;/blockquote&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;conda init&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;blockquote&gt;
  &lt;p&gt;If you’d prefer that conda’s base environment not be activated on startup, 
   set the auto_activate_base parameter to false:&lt;/p&gt;
&lt;/blockquote&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;conda config &lt;span class=&quot;nt&quot;&gt;--set&lt;/span&gt; auto_activate_base &lt;span class=&quot;nb&quot;&gt;false&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;
</content>
 </entry>
 
 <entry>
   <title>CLSP Cluster Notes | Day 1</title>
   <link href="https://chenyi00.github.io/2020/08/28/clsp-setup/"/>
   <updated>2020-08-28T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/08/28/clsp-setup</id>
   <content type="html">&lt;h3 id=&quot;notes-on-ssh-config&quot;&gt;Notes on SSH config&lt;/h3&gt;

&lt;p&gt;There was a problem I ran into on my Mac when composing the config file under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;~/.ssh/&lt;/code&gt;, where even though the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;User&lt;/code&gt; was specified for a particular &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Host&lt;/code&gt;, it still defaults to my local system username when logging in via terminal. To resolve the issue, I added the following to the config file:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;Host &lt;span class=&quot;k&quot;&gt;*&lt;/span&gt;
    User &amp;lt;remote_login_name&amp;gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;A solution, yet temporary. I am not sure what will happen if I try specifying a different username for a different host - would I still run into the same problem with the current default?
&lt;!--more--&gt;&lt;/p&gt;

&lt;h3 id=&quot;notes-on-tmux&quot;&gt;Notes on tmux&lt;/h3&gt;

&lt;p&gt;Being a noob, I had no idea where to find the name of my tmux session. Found this on stackoverflow:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-shell&quot; data-lang=&quot;shell&quot;&gt;tmux display-message &lt;span class=&quot;nt&quot;&gt;-p&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'#S'&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Taught me a lesson: always name a new session to avoid getting lost…&lt;/p&gt;
</content>
 </entry>
 
 <entry>
   <title>Hello World!</title>
   <link href="https://chenyi00.github.io/2020/08/24/hello-world/"/>
   <updated>2020-08-24T00:00:00+00:00</updated>
   <id>https://chenyi00.github.io/2020/08/24/hello-world</id>
   <content type="html">&lt;p&gt;It’s mandatory ;)&lt;/p&gt;
</content>
 </entry>
 

</feed>
