
Zipf's Law was proposed by the American linguist **George Kingsley Zipf** during the 1930s–1940s. It was initially formulated to describe the distribution of word frequencies in natural language: in a corpus, the frequency of a word is inversely proportional to its rank in the frequency table. Character or symbol sequences generally exhibit high entropy and often follow a long-tail distribution.

In other words:

- The most frequent word occurs the most times.
- The second most frequent word occurs roughly half as often as the first.
- The third most frequent word occurs approximately one-third as often as the first.
- And so on.

**Note:** Here, “word” refers to its base form (lemma or stem). This distinction is important for languages with rich inflection, such as Indo-European languages (e.g., English, German, Russian), where a single word can have multiple morphological variants. Zipf's Law measures the frequency of base tokens; without normalization, each morphological form would be treated as a separate token, which “splits” the frequency of high-frequency words and affects the slope calculation.

Zipf's Law is a typical **power-law distribution**. Mathematically, it can be expressed as:

$$
f(r) \propto \frac{1}{r^s}
$$

Where:

- $f(r)$ : frequency of the word with rank $r$  
- $r$ : rank in descending frequency (1 = most frequent)  
- $s$ : exponent, usually close to 1 in natural language  

When $s = 1$, the distribution is the classical Zipf distribution.

For a text containing $N$ words, the probability of the word with rank $r$ is:

$$
P(r) = \frac{\frac{1}{r^s}}{\sum_{k=1}^{V} \frac{1}{k^s}}
$$

Where:

- $V$ : vocabulary size (number of unique words)  
- The denominator is a normalization factor to ensure the probabilities sum to 1  

This implies that **higher-ranked words have higher probabilities**, and the decay of probability follows a power law.

For visualization in charts, the law can also be expressed in a log-linear form:

$$
\log f(r) = -s \log r + C
$$

From this form, we can observe:

- On a log-log plot, frequency $f(r)$ vs. rank $r$ is approximately **linear**
- Slope = $-s$

Zipf proposed the **Principle of Least Effort**:

1. **Speakers**: prefer using a limited set of high-frequency words to minimize effort.  
2. **Listeners**: prefer a rich vocabulary to reduce ambiguity.  
3. **Resulting balance**: a few high-frequency words + many low-frequency words → power-law distribution.

In essence, language emerges as a **balance between speaker and listener efficiency**.

Due to the above factors, natural language exhibits:

- **Few high-frequency words, many low-frequency words**
- **Long-tail effect**: most words occur only once or a few times
- **Scalability**: the larger the text, the more apparent the pattern

Zipf's Law informs multiple areas in **Natural Language Processing (NLP)**:

- TF-IDF weighting
- Search engine ranking
- Word frequency analysis
- Stopword selection
- Corpus construction
- Text compression

Fundamentally, Zipf's Law reflects a general **power-law phenomenon**, not limited to language.

1. **Short texts**: the law is less apparent in small corpora, as it is a macro-level statistical phenomenon.  
2. **Formal symbols**: some formal languages or programming code may exhibit similar distributions, but with more regular patterns; additional features are needed for interpretation.  
3. **Exponent $s$**: may vary around 0.9–1.1 depending on language and corpus, not necessarily exactly 1.

Zipf's Law reveals the **power-law distribution of word frequency** in natural language: few words are extremely frequent, many words are rare, forming a “long tail.” On a log-log plot, this relationship approximates a straight line. It serves as a foundational statistical principle in natural language, information retrieval, and social sciences.

Based on the knowledge of Zipf's law from this blog post, we can create a TypeScript program that roughly determines whether a text is a natural language or a formal language. The key idea is:

+ Count word frequencies to get the number of times each word appears.
+ Calculate an estimate of the Zipf index s, or at least determine whether the word frequency distribution has a long tail.
+ If the word frequency distribution in a text approximates a power-law distribution with a significant long tail → natural language
+ If most words have similar frequencies or a very regular distribution → formal language/code

Here’s an **enhanced TypeScript version** that combines both the **Zipf slope** and a **long-tail ratio** to make a more robust distinction between natural language and formal language/code, even for short texts:

```typescript
// enhanced-language-detector.ts

type WordFreq = Record<string, number>;

/**
 * Tokenize the text: split by non-word characters
 */
function tokenize(text: string): string[] {
  return text
    .toLowerCase()
    .replace(/[^a-zA-Z0-9_]+/g, ' ')
    .split(/\s+/)
    .filter(Boolean);
}

/**
 * Count word frequencies
 */
function countFrequencies(words: string[]): WordFreq {
  const freq: WordFreq = {};
  for (const word of words) {
    freq[word] = (freq[word] || 0) + 1;
  }
  return freq;
}

/**
 * Sort frequencies in descending order
 */
function sortFrequencies(freq: WordFreq): number[] {
  return Object.values(freq).sort((a, b) => b - a);
}

/**
 * Estimate the Zipf slope using log-log regression
 */
function estimateZipfSlope(freqs: number[]): number {
  const n = freqs.length;
  const ranks = freqs.map((_, i) => i + 1);
  const logF = freqs.map(f => Math.log(f));
  const logR = ranks.map(r => Math.log(r));

  const avgLogF = logF.reduce((a, b) => a + b, 0) / n;
  const avgLogR = logR.reduce((a, b) => a + b, 0) / n;

  let numerator = 0;
  let denominator = 0;
  for (let i = 0; i < n; i++) {
    numerator += (logR[i] - avgLogR) * (logF[i] - avgLogF);
    denominator += (logR[i] - avgLogR) ** 2;
  }

  return -numerator / denominator;
}

/**
 * Calculate long-tail ratio: proportion of words that appear only once
 */
function longTailRatio(freqs: number[]): number {
  const singletons = freqs.filter(f => f === 1).length;
  return singletons / freqs.length;
}

/**
 * Detect if text is natural language or formal language/code
 */
function detectLanguageType(text: string): string {
  const words = tokenize(text);
  if (words.length < 20) return 'Text too short to decide';

  const freqMap = countFrequencies(words);
  const freqs = sortFrequencies(freqMap);

  const slope = estimateZipfSlope(freqs);
  const tailRatio = longTailRatio(freqs);

  // Heuristic rules:
  // Natural language: slope ~0.9-1.1 and long-tail ratio > 0.4
  if (slope > 0.7 && slope < 1.3 && tailRatio > 0.4) {
    return `Probably natural language (Zipf slope ≈ ${slope.toFixed(2)}, long-tail ratio ≈ ${(
      tailRatio * 100
    ).toFixed(1)}%)`;
  } else {
    return `Probably formal language or code (Zipf slope ≈ ${slope.toFixed(2)}, long-tail ratio ≈ ${(
      tailRatio * 100
    ).toFixed(1)}%)`;
  }
}

// --------- Test ---------
const naturalText = `
Zipf's law was proposed by George Kingsley Zipf during the 1930s–1940s.
It describes word frequency distributions in natural language.
`;

const formalText = `
int main() {
  int a = 0;
  int b = 1;
  return a + b;
}
`;

console.log(detectLanguageType(naturalText)); // Probably natural language
console.log(detectLanguageType(formalText));  // Probably formal language
```



The ZIPF Statistical Theory for Symbol Text.

In natural language, the frequency of a word is inversely proportional to its rank in the word frequency. Symbol sequences usually have high entropy and a long-tail distribution.

How to Distinguish Natural Language and Formal Language


In frontend interaction between components and data state there are some exist and mature thoughts like One-way binding (e.g. React, Svelte) and Two-way binding(e.g. Vue).

Now we will peruse the two-way binding which makes the UI and state automatically update each other.

Here are the graph to differentiate them.

```dot
digraph Binding {
  rankdir=LR;
  node [shape=box, style=rounded];

  subgraph cluster_oneway {
    label="(One-Way Binding)";
    style=dashed;

    State1 [label="State"];
    UI1 [label="UI"];

    State1 -> UI1 [label="Data Flow"];
  }

  subgraph cluster_twoway {
    label="Two-Way Binding";
    style=dashed;

    State2 [label="State"];
    UI2 [label="UI"];

    State2 -> UI2 [label="View Update"];
    UI2 -> State2 [label="Control Flow"];
  }
}
```

## Event Listening + DOM

The earliest two-way binding was done purely manually. For example, for a text input: listen to the DOM's `input`/`change` event, manually update the data, and then manually modify the DOM.

```js
const data = { message: "Hello" };
const input = document.querySelector("#input");
const display = document.querySelector("#display");

// Data -> View
function updateView() {
  input.value = data.message;
  display.textContent = data.message;
}

// View -> Data
input.addEventListener("input", e => {
  data.message = e.target.value;
  updateView();
});

// Initialize
updateView();
```

This is the ancient implementation method, the most intuitive, compatible with all browsers, but cumbersome and high maintenance cost. When the complexity of frontend pages reaches a certain level, the maintenance cost increases exponentially.

## `defineProperty` Data Hijacking

To solve the complexity problem of the above maintenance cost, Vue2 proposed a simple method: utilize JavaScript language features, bind getters and setters, to achieve the effect of automatically updating the view and state bidirectionally.

Before `Proxy` appeared, the most common way was to use `Object.defineProperty` to intercept the object property's `getter / setter`, triggering view updates when setting properties.

The general idea is:

* Use `Object.defineProperty` to hijack the data's getter/setter.
* Collect dependencies in the getter (who uses this data).
* Notify dependencies to update in the setter (Publish-Subscribe pattern).

```js
const data = {};
let value = "Hello";

Object.defineProperty(data, "message", {
  get() {
    return value;
  },
  set(newVal) {
    value = newVal;
    console.log("View updated:", newVal);
    document.querySelector("#input").value = newVal;
  }
});

// Data -> View
data.message = "Hi defineProperty";

// View -> Data
document.querySelector("#input").addEventListener("input", e => {
  data.message = e.target.value;
});
```

This method has good compatibility in IE9+. But the disadvantage is: it can only hijack existing properties; adding/deleting properties cannot be listened to.

## Dirty Checking

Angular.js took a different approach: continuously polling the consistency of the state and view, and then updating the corresponding state and view. This is also called the **Dirty Checking mechanism**: regularly checking whether the data and DOM are consistent, and updating if they are not.

For Angular, its core is a mechanism called the `$digest` loop: whenever data might change (user input, click events, Ajax callbacks, `$timeout`, `$http`, etc., at appropriate times), Angular triggers a dirty check cycle.

During the `$digest` process, Angular checks all Watchers (each bound data corresponds to a watcher) in turn. If it finds that some data is different from the last value, it calls the corresponding callback to update the DOM.

Because updating one piece of data may trigger changes in other data, $digest will run up to 10 times (to prevent infinite loops).

For demonstration convenience, use `setInterval` to temporarily replace polling for dirty checking.

```js
let data = { message: "Hello" };
let oldMessage = data.message;

setInterval(() => {
  if (data.message !== oldMessage) {
    oldMessage = data.message;
    console.log("Update view:", data.message);
    document.querySelector("#display").textContent = data.message;
  }
  // #display is the input component
}, 100);
```

This approach is simple to implement. But the performance is very poor; when involving large-scale data state updates, obvious lag can be felt.

Therefore, in Angular 2.x and later versions, this "two-way binding" idea was abandoned. After using TypeScript, the design philosophy changed to unidirectional data flow + one-way binding. To achieve two-way binding, it must be implemented manually, and Angular also provides corresponding syntactic sugar.

## Proxy

Vue2 implemented reactivity based on `Object.defineProperty`, but there were mainly several problems:

1.  Can only hijack existing properties. You must first define the properties on the object, then use `Object.defineProperty` to add getter/setter. Moreover, operations like adding properties / deleting properties cannot be listened to (Vue2 needed `Vue.set`, `Vue.delete` hacks).
2.  Cannot directly listen to array index changes. Methods like `push/pop/shift/unshift/splice` on arrays need to be rewritten separately. And operations like direct assignment are also not friendly; directly doing `arr[0] = newVal` cannot trigger updates.
3.  Requires recursively traversing the object. Need to add getter/setter to every property at every level of the listened object. However, initializing deeply nested objects is very performance-intensive.

After the ES6 standard was released, Vue3 used `Proxy` to implement reactivity (replacing Vue2's `Object.defineProperty`). Proxy is a relatively new and elegant approach.

Vue3 rewrote the reactivity system using `Proxy`, solving the above problems. The `getter/setter` of the Proxy method is somewhat different from the `getter/setter` set by `Object.defineProperty`. Specifically:

1.  **Can intercept the entire object** `Proxy` does not need to define getter/setter for each property individually. It can be handled uniformly directly in `get/set`.
2.  **Supports dynamic properties** Adding/deleting properties can be intercepted and listened to. So extra APIs like `Vue.set` are no longer needed.
3.  **Native array support** Accessing array indices, `length`, or executing methods like `push` will trigger the `set` interceptor. Therefore, there's no need to manually rewrite the array prototype for extra array operations.
4.  **Supports more operations** `Object.defineProperty` can only intercept property read/write. But `Proxy` can intercept 13 operations: `get`, `set`, `deleteProperty`, etc. This makes the reactivity system more flexible (e.g., `for...in` traversal can also trigger dependency collection).

Proxy can also achieve **better performance**. When initializing an object, there's no need to deeply traverse all properties; it only triggers `get` when actually accessed, resulting in better performance.

Next, let's use the Proxy mechanism to implement a **real DOM two-way binding version** that can bind object properties to `<input>` elements, using the Proxy mechanism to implement a simple two-way binding mechanism.

```html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Proxy Two-Way Binding Example</title>
</head>
<body>
  <input id="input" type="text" />
  <p id="display"></p>

  <script>
    // Data object
    const data = { message: 'Hello Proxy' };

    // Get DOM elements
    const inputEl = document.getElementById('input');
    const displayEl = document.getElementById('display');

    // Display initial value
    inputEl.value = data.message;
    displayEl.textContent = data.message;

    // Create Proxy, implement Data -> View
    const proxyData = new Proxy(data, {
      set(target, prop, value) {
        target[prop] = value;

        // Update view when data changes
        if (prop === 'message') {
          inputEl.value = value;
          displayEl.textContent = value;
          console.log(`View updated: ${prop} = ${value}`);
        }

        return true;
      },
      get(target, prop) {
        return target[prop];
      }
    });

    // View -> Data binding
    inputEl.addEventListener('input', (e) => {
      proxyData.message = e.target.value; // Automatically update data, trigger Proxy
      console.log(`Input changed: ${e.target.value}`);
    });

    // Test
    setTimeout(() => {
      proxyData.message = 'Hello Two-Way Binding!';
    }, 2000);
  </script>
</body>
</html>
```

In the above code,

1.  When the data changes, automatically update the `<input>` and `<p>` display.
2.  When the input box changes, automatically update the data object.
3.  The `set` interceptor of `Proxy` is responsible for Data -> View, the `input` event is responsible for View -> Data.


Two-ways binding's minimal implementations and simple examples with JavaScript in frontend.

Two-Ways Binding in Frontend Implementation


The **`WeakMap`** is a special type of collection in JavaScript. It was formally introduced in the ES6 standard and is similar to `Map`, but with several key differences:

- A `WeakMap` is a collection of **key-value pairs (key → value)**;
- **Keys can only be objects** (plain objects, arrays, functions, etc.), **not primitives** (numbers, strings, booleans, etc.);
- `WeakMap` holds **weak references** to its keys, meaning:
  If no other variables reference the object anymore, the garbage collector will automatically reclaim it, and the corresponding entry in the `WeakMap` will disappear.

| Feature       | `Map`                                                   | `WeakMap`                                           |
| ------------- | ------------------------------------------------------- | --------------------------------------------------- |
| Key Types     | Any value (objects or primitives)                       | Objects only                                        |
| Key Reference | Strong reference                                        | Weak reference                                      |
| Enumerable    | Enumerable (iterable via `map.keys()`, `map.entries()`) | Not enumerable (no iteration methods)               |
| Use Cases     | General key-value mapping storage                       | Data that needs to be tied to an object's lifecycle |

Why are weak references needed? Suppose we want to attach "extra information" to certain objects but don't want this information to prevent the objects from being garbage collected. For example:

```js
const wm = new WeakMap();

function createUser(name) {
  const user = { name };
  wm.set(user, { created: Date.now() });
  return user;
}

let alice = createUser("Alice");
console.log(wm.get(alice));
// { created: 1692624526805 }  // e.g., stores creation time

// When alice is no longer referenced, the object will be GC'd, and the entry in the WeakMap disappears
alice = null;
```

Here, the role of `WeakMap` is to store **additional metadata** without worrying about the object "living forever" because it's stored in a Map.

`WeakMap` acts like an **Object Additional Information Warehouse** —the keys inside it do not affect object reclamation. Its greatest value is: **following the object's lifecycle and avoiding memory leaks**.

Many people mistakenly think that the "weak" reference in **WeakMap is implemented via reference counting**, but that's not the case.

**Reference counting** is a garbage collection algorithm: an object's reference count drops to zero → memory is released. However, modern JavaScript engines (V8, SpiderMonkey, etc.) **do not rely solely on reference counting**. Instead, they use garbage collection centered around **reachability analysis**. Therefore, the "weak reference" in WeakMap is **not "delete when reference count = 0"**. Rather:
**The key in a WeakMap is not considered a "root of reachability"**.

In a regular `Map`, the key object is strongly referenced. Even if no external variables reference it anymore, it remains "reachable" and won't be GC'd.

In a `WeakMap`, the key is a **weak reference**. The garbage collector does not count it as part of the "live objects": if there are no other strong references from the outside, the key will be reclaimed by GC, and the corresponding key → value mapping will disappear.

Because the JS engine needs the freedom to remove entries during GC, if it allowed you to enumerate the WeakMap, you might see entries that "have been reclaimed but not yet cleaned up," making the semantics uncertain.
Therefore, WeakMap only supports `set/get/has/delete` and **cannot be iterated**.

The underlying mechanism of WeakMap is not "reference counting" but is based on the **reachability-based garbage collection mechanism**. Its "weak reference" essentially tells the GC: **Keys in the WeakMap do not count as active references**. Thus, WeakMap is very suitable for storing "additional data that disappears along with the object."

Below is a diagram illustrating the difference in garbage collection between a **regular Map and a WeakMap**.

For a strongly referencing Map:

```js
let map = new Map();
let obj = { name: "Alice" };

map.set(obj, "metadata");

// Now, even if obj = null,
// the map still "strongly references" this object
obj = null;
```

The memory relationship is:

```
[Global variable map] ──► [Map instance]
                               │
                               ▼
                         ┌───────────────┐
                         │ key: {Alice}  │──► "metadata"
                         └───────────────┘
```

Even after `obj = null`, the Map still holds `{Alice}`, which won't be GC'd. **This can easily cause memory leaks**.

For a weakly referencing WeakMap:

```js
let wm = new WeakMap();
let obj = { name: "Alice" };

wm.set(obj, "metadata");

// Remove the external reference to obj
obj = null;
```

Its memory relationship is as follows:

```
[Global variable wm] ──► [WeakMap instance]
                                 │
                                 ▼
                           ┌───────────────┐
                           │ key: {Alice}  │──► "metadata"
                           └───────────────┘
```

After `obj = null`, there are no strong references pointing to `{Alice}`: GC considers `{Alice}` "unreachable," reclaims the object, and the key-value pair in the WeakMap automatically disappears.

This **does not prevent garbage collection**.

In Node.js, directly observing garbage collection is difficult because **GC is automatic**.
However, starting from Node.js v14, `WeakRef` and `FinalizationRegistry` are available, allowing us to **simulate the WeakMap reclamation process**.

Here is some experimental code:

```js
// Save a WeakMap
const wm = new WeakMap();

// Register a FinalizationRegistry to notify when an object is reclaimed
const registry = new FinalizationRegistry((heldValue) => {
  console.log(`Object has been reclaimed, the corresponding metadata was: ${heldValue}`);
});

(function test() {
  let obj = { name: "Alice" };

  // Store a record in the WeakMap
  wm.set(obj, "metadata for Alice");

  // Register a reclamation callback for the object
  registry.register(obj, "metadata for Alice");

  console.log("Object created and stored in WeakMap");

  // Manually clear the strong reference to obj
  obj = null;
})();

// Prompt Node.js to perform garbage collection
// ⚠️ Requires running with the flag: node --expose-gc test.js
setTimeout(() => {
  console.log("Attempting to manually trigger garbage collection");
  if (global.gc) {
    global.gc();
  } else {
    console.log("Please run this script with node --expose-gc");
  }
}, 1000);
```

Next, save the code as `test.js` and run it with the GC flag:

```bash
node --expose-gc test.js
```
You will see output similar to:

```
Object created and stored in WeakMap
Attempting to manually trigger garbage collection
Object has been reclaimed, the corresponding metadata was: metadata for Alice
```

Thus, we can see:

* The key in `WeakMap` does not prevent object reclamation;
* `FinalizationRegistry` can trigger a callback after the object is GC'd, proving the WeakMap entry has disappeared;
* This shows the essence of WeakMap's "weak reference" is: **It does not treat the key as a root of reachability for GC**.

Below are several **typical use cases for WeakMap**. You'll find it's primarily used for "attaching extra information to objects without wanting to affect the object's lifecycle":

1.  **Private properties** (security + automatic release)
2.  **Caching** (automatically cleaned up when the object is released)
3.  **DOM/object metadata storage** (without polluting the original object)
4.  **Event listener management**
5.  **Implicit information attachment inside libraries**

**Private property storage** (simulating class private fields)

```js
const privateData = new WeakMap();

class User {
  constructor(name, password) {
    this.name = name;
    privateData.set(this, { password });
  }

  checkPassword(pw) {
    return privateData.get(this).password === pw;
  }
}

const u = new User("Alice", "123456");
console.log(u.checkPassword("123456")); // true
console.log(u.password); // undefined (cannot access directly)
```

Using WeakMap to save "private data"; when the object instance is reclaimed, the data is automatically released too.

**Caching (Memoization)**

```js
const cache = new WeakMap();

function heavyCompute(obj) {
  if (cache.has(obj)) {
    return cache.get(obj);
  }
  let result = expensiveCalculation(obj);
  cache.set(obj, result);
  return result;
}

function expensiveCalculation(o) {
  return JSON.stringify(o).length; // Assume it's computationally expensive
}

let data = { foo: "bar" };
console.log(heavyCompute(data)); // Compute and cache
console.log(heavyCompute(data)); // Read directly from cache
data = null; // Release the object, cache automatically cleans up
```

Ideal for "object → result" caching scenarios, prevents memory leaks.

**DOM metadata storage**

```js
const elementData = new WeakMap();

function bindInfo(el, info) {
  elementData.set(el, info);
}

function getInfo(el) {
  return elementData.get(el);
}

// Usage
const button = document.querySelector("button");
bindInfo(button, { clicked: 0 });

button.addEventListener("click", () => {
  const info = getInfo(button);
  info.clicked++;
  console.log(`Button click count: ${info.clicked}`);
});
```

When the `button` node is removed from the DOM and becomes unreachable, the WeakMap automatically clears the metadata, preventing memory leaks.

**Event listener state tracking**

```js
const listenerMap = new WeakMap();

function trackListener(obj, listener) {
  obj.addEventListener("click", listener);
  listenerMap.set(obj, listener);
}

function removeListener(obj) {
  const listener = listenerMap.get(obj);
  obj.removeEventListener("click", listener);
  listenerMap.delete(obj);
}
```

Tracks bound listeners and automatically releases them.

**Cross-library/cross-module additional information**

If you are writing a library and don't want to pollute user objects (avoid adding properties directly), WeakMap can "attach" data externally:

```js
// myLib.js
const meta = new WeakMap();

export function attach(obj, info) {
  meta.set(obj, info);
}

export function getInfo(obj) {
  return meta.get(obj);
}
```

Users won't see extra properties, and there's no risk of memory leaks.


The introduction and usage examples for Weakmap in ES6, with comparing to the Map.

ES6 Note: Weakmap


我们都应该为身边有这样的人，痛苦地活着，又痛苦地死去，而感到惭愧。🕯️

![](https://image-assets.dreams.plus/202507071733397.png)


我们都应该为身边有这样的人，痛苦地活着，又痛苦地死去，而感到惭愧。

心非木石岂无感


Internationalization (i18n) and localization (l10n) involve numerous intricate details. At the beginning I just want to draft a checklist for i18n & l10n design. This checklist serves as a reference for planning during the software design phase. However, it provides only a high-level overview. Since different countries and regions have unique requirements and standards, this document does not cover specific implementation details. Instead, it highlights key considerations to keep in mind when designing for international audiences.

> This document is a work in progress. And I have opensourced this draft on github: https://github.com/PrinOrange/i18n-l10n-checklist.git  

## Language & Translation  

### Character Sets and Encoding 

Although UTF-8 is widely adopted, some platforms may not support it by default and instead use their own encoding schemes, such as ISO 8859-1, GBK, JIS, or Big5.

Have you accounted for this in your application's character processing? Does your platform support displaying the required character sets, including Latin, CJK ideographic, Cyrillic, Thai, and others?

Here is the wikipedia page for [charset encoding](https://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings). 

### Transcription Systems  

In cases where certain character sets are unsupported, do you need a transcription system? For example, some Unix-based systems only support the ASCII character set and cannot display Cyrillic characters, requiring transliteration into Latin script.  

### Dialects and Regional Variants  

+ **Multi Official Language:** A country or region may have multiple official languages or dialects. For example, Singapore recognizes English, Chinese, and Malay, while Hong Kong uses both Cantonese and English. Have you accounted for these linguistic variations?  
+ **Region dialects:** Additionally, a single language may have multiple regional variations. For instance, Arabic as spoken in Saudi Arabia differs significantly from that used in the United Arab Emirates. Likewise, spelling and pronunciation vary between British English, Australian English, and American English.

### Fonts

For some languages use same character set may have different font glyphs.

- Japanese Kanji and Chinese character has different glyph standard. For same character, Chinese font and Japanese font may has different displaying formals.
- Vietnamese and English share Latin characters, but Vietnamese has tone annotations, so the center of gravity in Vietnamese fonts is lower than that of English.

## UI, Layout & Typography  

### Right-to-Left (RTL) & Left-to-Right (LTR) Support  

Have you ensured that your UI properly supports RTL languages such as Arabic and Hebrew? 

+ In RTL layout, the position of default button, visual direction are reverse.
+ Some animation directions are reverse, for example, progress bar, switch button and carousel slides.
+ Sequence logic are reverse. For example, the `Next` button is on left while `Previous` button is on right. And the pagination buttons are reverse too.
+ Some icons should not be reverse, like clock, handwriting icons.
+ For mathematic formulas, some Arabic region use latin-form formula while other use traditional Arabic numeral notations form, this should be reverse written, like. [See formula directions in Arabic document.](https://en.wikipedia.org/wiki/Modern_Arabic_mathematical_notation)

### Line Breaking & Layout Adjustments  

Different languages have different line-breaking rules. For example, Chinese and Japanese do not use spaces to separate words, while Thai requires special segmentation rules. Does your layout handle these differences correctly without causing layout thrashing?  

## UX & Interaction Design  

### Keyboard Layouts

Keyboard layouts vary by region. Common layouts include QWERTY, QWERTZ, and AZERTY, and even within the same language, there may be differences. For example, the U.S. uses a standard QWERTY layout, while the U.K. has a slightly different variation. [See more details](https://en.wikipedia.org/wiki/Keyboard_layout).  

+ When designing shortcut keys, ensure they remain accessible and intuitive across different keyboard layouts.
+ Have your software been adopted for user's language Input Method Editor (IME). You need to ensure that the user's input method can be worked normally in your software.

## Content & Resources  

### Sorting Rules  

Sorting rules vary between regions, even though for one language. 

- Most of regions that uses alphabets-based language typically use alphabet-dictionary order based on item titles.
- In mainland China, sorting by Pinyin is common. Hong Kong sorts by stroke count, whereas Taiwan follows Bopomofo order.
- Note: some character sorting in UTF encoding are not equal to them in alphabets for some languages. For example, the Finnish letters `Å, Ä, Ö` are ranked last in the alphabet (after Z) in Swedish, Finnish, Estonian and other languages. However, the default sorting of computers (such as strcmp or JavaScript localeCompare in C) may be sorted by Unicode code points: `A, Å, Ä, B, C, ..., O, Ö, P, Q, ..., Z`

Moreover, alphabetical orders differ across languages, affecting sorting logic. Ensure your sorting algorithm accounts for these regional differences.  

### Date Formats, Time Zones & Daylight Saving Time  

+ **Date format:** Which date format is used in your target regions? Is it *DD/MM/YYYY*, *YYYY/MM/DD*, or *MM/DD/YYYY*? [See the distribution map of date formatting among different regions](https://en.wikipedia.org/wiki/List_of_date_formats_by_country).
+ **Time Zone:** Have you considered the time zone differences for each regions and countries? For example, the U.S. actually has six time zones, Eastern Time Zone, Central Time Zone, Mountain Time Zone, Pacific Time Zone, Alaska Time Zone, and Hawaii-Aleutian Time Zone. Here is the [details for time zone map](https://www.timeanddate.com/time/map/).
+ **Daylight saving time:** Are month names written as numbers or spelled out? Do you need to account for daylight saving time adjustments? For example, some European region and U.S. have daylight saving time.
+ **Special Calendar:** Some countries may not use the Gregorian calendar. For example, Japan uses the Era-Name Calendar(令和/平成/昭和), and Thailand uses the Thai Solar Calendar.

### Number Formats, Measurement Units & Mathematical Conventions  

Different regions have unique numerical formatting and measurement units.  

- The U.S. and U.K. use a comma (`1,000,000`) as a thousand separator and a dot (`1.5`) for decimals, whereas Germany, France and other European countries reverse this notation (`1.000.000` and `1,5`).  
- India groups numbers differently, using a **2-2-3** format: `10,00,000` instead of `1,000,000`.  
- Measurement units vary: the U.S. uses inches, feet, and pounds, while most of the world follows the metric system.  
- Not every regions only use Western Arabic numerals (like $0,1,2,3,...,9$), for middle eastern regions, they prefer using traditional eastern Arabic numerals(like 0-9, their written form are separately like $٠-0,١-1,٢-2,٣-3,٤-4,٥-5,٦-6,٧-7,٨-8,٩-9$)
- Numbering rules. For some situation the numbering rules depends on formatting rules, like chapter numbering, they prefer using Roman numerals like `Chapter Ⅰ`, `Chapter Ⅳ`.

Ensure your application can handle these variations dynamically.  

### Natural Language Processing (NLP) Considerations  

If your application involves NLP features such as search, spell checking, or text analysis, are these adapted for different languages?

+ Tokenizer design: for different language there are variant ways to process tokens and words during text indexing for search feature. Your should take intrinsic details of different languages into consideration, like stemmer, removing stopwords.
+ Declension. In i18n, it is necessary to deal with word inflection, because the same word may change depending on the context when expressing grammatical structures such as number, gender, tense, case, etc, especially in situation about string concat. If these inflections are not handled, the translated content may be grammatically incorrect, unnatural, or even difficult to understand, which will seriously affect the user experience and product professionalism.

### Audio, Video & Media Localization  

Have you prepared subtitles and translated resources for different languages? If your product supports live streaming or real-time interactions, do you provide interpretation services?  

### Cultural Sensitivities for Symbolism  

Have you accounted for cultural taboos and regional sensitivities?  

- In monarchies like Thailand and Iran, criticism of the royal family is strictly prohibited.  
- In Muslim-majority countries, certain animal symbols (e.g., pigs, donkeys) are considered inappropriate.  
- Some regions restrict religious imagery; for example, Christmas-related elements are not permitted in Saudi Arabia and Yemen.  
- Symbols can have vastly different meanings:  
  - The “OK” gesture (`👌`) is positive in the U.S. but offensive in France, Germany, and Brazil.  
  - The thumbs-up sign (`👍`) may be considered sarcastic in Greece and parts of Latin America.  

To avoid unintended offense, research regional interpretations before including symbols in your UI.  

## Monetization & Payments  

+ **Currency Unit:** What is the smallest currency unit in the target market? In China, it's 0.1 yuan, whereas in the U.S., it's 0.01 dollars.  
+ **Regulation:** Are currency exchange rates, transaction policies, and regional banking regulations accounted for?  
+ **Formal Written:** How are large amounts displayed? In China, Japan and Korea, amounts may be written in uppercase Chinese characters(壹/贰/叁/肆/伍), whereas English uses full word spelling, for example, the number `1,013,978.1` is written in form "One million thirteen thousand nine hundred seventy-eight and one tenth".

## Performance & Optimization  

### CDN & Network Zones  

Have you optimized content delivery networks (CDNs) for different regions to ensure low-latency performance? Is it optimized for local networks?

- Fallback Logic: Handle missing locales with graceful fallback, for example, the Mexico speaks spanish and it's supposed to load language resource `es-MX` but if there are missing such resource, you could make fallback logic, even fallback to default language: `es-MX → es → en`
- Avoid Query Strings for Locale: Query strings may break caching. Prefer path-based routing.
- Language-based URL Structure: Use paths like /i18n/en-US.json instead of query params for better caching.
- Lazy Load Translations: Only load the needed language file (e.g., fr-FR.json) at runtime.
- Font & Asset Localization: Split requests of fonts/images by language — especially for large scripts like Chinese, Japanese.

## Security & Access Control  

Are there specific compliance or data protection laws that need to be followed, such as GDPR in Europe or PIPL in China?

+ You should seek for user's permission according to the law the *ePrivacy Directive* by setting cookie acceptance button for websites in European and America.

## Legal & Compliance  

Have you considered regional legal requirements related to software distribution, content moderation, and user privacy? For example, 

+ The European Union has very strict privacy legislation. If you collect user's interactions and information without permission, you will be fined.
+ In China, your software and website must be verified with ICP and copyright license before publishing, and annotate the license notably in specified position.

## Accessibility & Inclusive Design  

Have you accounted for accessibility standards?  

- Different countries use different Braille systems and sign languages.  
- Screen readers and assistive technologies should be tested across various locales. 

## Cultural Factor

Certain expressions or phrases may have region-specific variations that need careful translation to avoid sounding unnatural or inappropriate. 

+ For example, in Arabic-speaking regions, formal texts often include religious phrases like "Allahu Akbar" ("Allah is great"). 
+ In Japan and South Korea, appropriate honorifics should be used based on social and contextual factors, like「です」「ます」, and「습니다」 

## References  

- [W3C Internationalization Best Practices](https://w3c.github.io/bp-i18n-specdev/)  


The Checklist for software i18n design. This checklist is intended to provide a checklist for planning reference during the software design phase. Of course, this is just an outline.

Drafts for I18n & L10n Mind Framework.


> **NOTE:** This paper's implementation in TypeScript Programming is published on Github, repo: https://github.com/PrinOrange/ts-ngcf , and paper: https://dl.acm.org/doi/10.1145/3331184.3331267

## Key Conceptions
+ **Collaborative Filtering**: It denotes the implicit group-behavior patterns for interactions between users and objects(for example, clicking, voting and purchasing). It's used for speculating user's preferences and correlations among objects. It assumed that behaviorally similar users would have similar preferences on items. CF has two key components, embedding and modeling.
+ **Interaction modeling**: A progress of reconstructing historical interactions based on the embeddings, the interaction function details could be based on factor matrix, non-linear neural networks and Euclidean distance metrics. 
+ **Embedding**: Learning vector representations of users and items. For convenience, we vectorize the features from users and items, and transform them into vector form.
+ **Collaborative signal**: The implicit correlations between users and objects. It indicates where are the user's interests in between objects and their behaviors. 
+ **bipartite graph**: It's a special graph structure whose nodes can be divided into two non-intersecting sets, and each edge in the graph can only connect nodes in these two sets, while there is no direct connection between nodes in the same set.
+ **High-order connectivity**: refers to multi-hop interaction paths (such as user A → item 1 → user B → item 2), which can reflect potential interest associations, `user A` might be interested in `item 2`
+ **Decay factor**: The decay factor is an important hyperparameter used to control the attenuation of the influence of neighbor nodes on the target node as the number of path hops increases during information propagation. Its core function is to balance the weights of local information and global information and avoid noise interference from high-order neighbors.
+ **LeakyReLU** is an activation function. It is an improved version of ReLU (Rectified Linear Unit) and is used to solve the "dead neurons" problem that may occur during ReLU training.
+ **element-wise product**: In the recommendation system, it is used to represent the relationship between user and item embedding
+ **BPR Loss**: It's a loss function used for description of loss in recommendation to compare the priority between two items.

## What problem does this paper solve.
Traditionally the recommendation methodology uses the matrix and deep learning to take user's attributions and item's inherent features into consideration, but it has defects that it ignored the implicitly latent factors(namely collaborative signal) during encoding progress. 

For example, there is a pregnant user and she wants to purchase a milk powder, the tradition methodology tags this user with `baby`, `pregnant`, `infant`, while the goods of milk powder are tagged with `baby food`. Then the recommendation system simply could only capture that tags `baby`, `infant` from user and tags `baby food` from milk powder are in one class, and recommend milk powder list to user. The system matches these tags directly (`baby` <-> `baby food`) and recommends milk powder because the tags are explicit factor.

But it ignored that a pregnant user not only needs the milk powder simply but also other relevant goods, she may be need milk bottle and nipple after she brought the milk powder for her baby.

So there is the implicitly latent correlation between "purchasing milk powder" and "milk bottle", "nipple". That is namely collaborative signal.

The traditional methodology has such defect and that is what's the paper focusing on. This paper prompt a new idea that integrates user-item interactions as collaborative signal into embedding process with structure named `bipartite graph `, and it develop a new framework named `Neural Graph Collaborative Filtering` which could exploits these latent factors and establish high-order connectivities between users and items with such data structure, and effectively inject the collaborative signals into embedding process.

The authors occurs that some methods for interaction modeling, like MF, nonlinear neural network, are not enough for embedding for they lacks the encoding for collaborative signals that could not take advantages of implicit interest correlations from behaviors.

Also, the authors devised many benchmark tests over this framework and verified its rationalities and improvements.

## How to embark on
The difficulty authors faced on is that information system may have large-scale dataset so that its hard to distill and refine the collaborative signal. So the authors deal with this problem by exploiting high-order connections from user-item reactions with encoding collaborative signal into reaction graph.

The authors are inspired by recent works of graph neural networks, and compared existing HOP-Rec to NGCF, pointed out that HOP-Rec still be based on MF thar can not enough exploit training data. NGCF incorporates high-order connectivity directly into the model architecture and dynamically adjust the embedding representation through embedding propagation to generate better embeddings.

![](https://image-assets.dreams.plus/images202504081104541.png)

The _figure 1_ indicates how and what information does this graph provides. There are lines connect $(u_{1})$ to $(i_{1})$, $(i_{2})$ and $(i_{3})$ and it means there are semantically behavioral relationships between user $(u_{1})$ with $(i_{1})$, $(i_{2})$ and $(i_{3})$

Recommendation process for user $(u_{1})$:

+ First-level propagation: $(u_{1} \leftrightarrow i_{2})$ (direct interaction).
+ Second-level propagation: $(u_{1} 	\leftrightarrow i_{2} 	\leftrightarrow u_{2})$ (discovering similar behavior to $(u_{2})$).
+ Third-level propagation: $(u_{1} 	\leftrightarrow i_{2} 	\leftrightarrow u_{2} 	\leftrightarrow i_{4})$ (inferring that $(u_{1})$ may like $(i_{4})$)

The model also decides which of $(i_{4})$ and $(i_{5})$ is more suitable to recommend to $(u_{1})$ through the learned information flow weights. Obviously, the count of paths connected $(u_{1})$ to $(i_{4})$ are more than $(i_{5})$.

More details, see page 2, Running Example and Present Work section.

## Methodology
The methodology of NGCF contains three lines:

+ Initialize embedding layer for user and item.
+ Multi embedding propagation layers to refine embeddings by injecting high-order connectivity.
+ Prediction layer to aggregate refined embeddings and output final affinities score for user-item tuple.

## First-order propagation
The users that consume an item can be treated as the item's features and used to measure the collaborative similarity of two items. NGCF designs an interaction-aware message construction mechanism that incorporates personalized user-item affinity (implemented through element-by-element multiplication) into the propagation process of graph neural networks, thereby generating more accurate embedding representations.

Each propagation includes two links: message construction and message aggregation.

**Message Construction**: for a connected user-item pair $((u,i))$, we define the message from $(i)$ to $(u)$ as:

$(m_{u←i} = f(e_{i} ,e_{u} ,p_{ui}),)$

where $(m_{u←i})$ is the message embedding. $(f)$ is the message encoding function, which takes embeddings $(e_{i})$ and $(e_{u})$ as input, and uses the coefficient $(p_{ui})$ to control the decay factor on each propagation on edge $((u,i))$.

How do the authors implement $(f)$, see the normalized formula $((3))$

This message is the unit of propagation in the graph neural network.

**Message Aggregation** Aggregate $(u)$'s propagations from neighborhoods and refine $(u)$'s representation, denoted with $(e^{(1)}_{u})$, its definition, see formula $((4))$, analogously, we also could denote item's representation by $(e^{(1)}_{i})$



## High-order propagation
After first-order propagation we also could stack more propagation layers to explore high-order connectivities and exploit more collaborative signals.

The $(l)$-th order propagation for user representation was denoted by symbol $(e^{(l)}_{u})$, analogously, $(e^{(l)}_{i})$ for $(l)$-th propagation for item. See formula $((5))$.

Also we could define the $(l)$-th propagation message as $(m^{(l)}_{u\leftarrow i})$ and $(m^{(l)}_{u\leftarrow u})$. See formula $((6))$.

Formula $((5))$ and $((6))$ are also could be transcript into formula $((7))$, in matrix form.

## Model Prediction
After we solve $(l)$-th layers and propagations we could get final representations for user and item, denoted by $(\{ e^{(0)}_{u}, ..., e^{(l)}_{u} \})$ and $(\{ e^{(0)}_{i}, ..., e^{(l)}_{i} \})$, and concat them respectively. The parameter $(l)$ is used for controlling the layers count and determine how spread the recommendation does.

Finally we conduct the preferences between user and item with such formula towards target item.

$$
({\hat{y}}_{NGCF}={e^{*}_{u}}^\top{e^{*}_{i}})
$$

And the caculation result is a scalar, indicates the user's preference to specified item in numberic score form. So as we can ranking items according to score number.

## Optimization
The author thought that BPR loss used in personalized recommendation systems has rooms for improvement, which aims to optimize the ranking of items through pairwise comparison. Its core idea is that for a certain user, the observed interaction items (such as clicks, purchases) should get higher prediction scores than the unobserved items. And it defined the loss function at formula $((11))$

By the way the author compared NGCF to MF based recommendation system, NGCF has two extra optimized advantages,

+ **Lower model size**: It only depends on embedding look-up matrix $(E^{(0)})$that transformed from user-item graph and weight matrix, with only $(2Ld_{l}d_{l-1})$parameters, is negligible compared to MD based model.
+ **Node and message dropout**: The authors refer the  node and message dropout in traning stage that randomly reduce outgoing message to avoid overfitting.



## Model Degradation
In discussion, the author pointed out that NGCF virtually is a expansion of traditional recommendation model SVD++ and NGCF. Including traditional models such as SVD++ and FISM in the same system showed that they were special cases of NGCF. SVD++ and FISM only utilize first-order interactions, while NGCF captures high-order cooperation signals through multi layers propagation, so it has stronger expressive power. Namely, NGCF is a general framework that can cover traditional models such as SVD++ and FISM by adjusting the number of propagation layers, message weights, and transformation functions.

**Downgrade to FISM**: in formula NGCF-SVD , set $(p_{iu'} = 0)$, namely prevent connection between user and items, NGCF has same behavior with FISM.

**Downgrade to SVD++**: if higher-order propagation is prohibited, multiple layers are not stacked, and only direct neighbor with first-order interaction information is retained, then NGCF would be downgraded to SVD++.



## Time Complexity
This model involved matrix multiplication, for $(l)$-th propagation, it needs triple-order matrix multiplication, with sizes of $(|R^{+}|)$, $(d_{l})$, $(d_{l-1})$

Also in $(l)$-th traning stage is $(|R^{+}|d_{l})$.

The summary of time optimization for whole procession is 

$$
(O\left(\sum_{l=1}^{L}|\mathcal{R}^{+}|d_{l}d_{l-1} + \sum_{l=1}^{L}|\mathcal{R}^{+}|d_{l}\right))
$$


## MF & Graph based model
It discusses two major problems of modern recommender systems:

1. **Traditional methods** such as matrix decomposition uses inner products to model user-item interactions, but their linear capabilities are limited;

2. **Deep learning methods**: such as NeuMF use neural networks to capture nonlinear interactions, but still **implicitly learn** collaborative filtering signals and cannot guarantee behavioral transitivity (such as indirectly related users/items are close in embeddings).

current models lack explicit encoding process of collaborative filtering signals (such as transitivity), resulting in poor embedding quality.

Briefly speaking, a better recommendation systems relied on implicit learning of user-item relationships, and need to explicitly model behavioral transitivity to improve embedding effects.

Recent years, research on recommendation system based on user-item interaction graphs has made some progress. early methods such as ItemRank and BiRank used label propagation technology to infer user preferences through node similarity in the graph structure. 

These methods are essentially neighborhood-based recommendations, but due to the lack of optimizable model parameters, their performance is often inferior to that of embedding-based models, to overcome this limitation, new methods such as HOP-Rec combine high-order connectivity information obtained by random walks with matrix decomposition and train recommendation models through the BPR objective function. It have shown that this method combined with graph structure can effectively improve the recommendation effect. However, existing methods still have obviously shortcomings: on the one hand, high-order connectivity is only used for data enhancement and is not directly integrated into the embedding function, on the other hand, the effect of random walks is highly dependent on the tuning of hyperparameters such as the attenuation factor, which makes the stability of the model challenging. Overall, current research has not fully explored the potential of graph structure information. 

How to directly encode high-order relationships into the embedding learning process is still a key issue that needs to be solved.

## Experiments
Authors perform experiments on three real-world datasets to evaluate our proposed method, especially the embedding propagation layer. And they aim to answer the following research questions: 

+ How does NGCF perform as compared with latest CF methods? 
+ How do different hyper-parameter settings, like depth of layer, embedding propagation layer, layer-aggregation mechanism, message dropout, and node dropout affect NGCF? 
+ How do the representations benefit from the high-order connectivity? 


This is the reading notes for NGCF Paper(arXiv:1905.08108), which propose to integrate the user-item interactions, more specifically the bipartite graph structure, into the embedding process, and exploits the user-item graph structure by propagating embeddings on it.

Neural Graph Collaborative Filtering


2023 年，那时候「两耳不闻窗外事，一心只读圣贤书」，在春招和秋招最激烈的时候我选择了们在图书馆里背书做题。大三暑假，我选择了留校备考。

2024 春，放榜时，我特意守在图书馆座位中，静待通知的那一刻出来。就当我满心欢喜迎接时，迎来的是一盆冷水。

当我看到惨不忍睹的成绩后，全身是懵的，虽然在此之前多次祈求上天垂幸，心存侥幸，然而现实仍给我当头一棒。或许是受到的刺激太大，连续几天身心全是木木的，血红色的分数在我梦魇中挥之不去。终于，心里慢慢接受了这个事实，星夜中独坐房台放声痛哭。

泪光中回顾了我一年苦逼的备考历程。炎炎夏日下在充斥汗味图书馆内做题，深冬腊月中抚着冻僵的手夜读，历历在目，无不诉说着心酸。可是为什么会是这种结果。。。

事后也算总结了教训，由于我过于乐观及对考研本身无经验，一战时发挥失利被刷。

当然，因为考研之事，自然也错过了春招秋招。校招，本是找工作的好机会，被我错过，简历投了上千家公司，几乎无一问津。

身边同学交集也逐渐变少，大四一年，每个人都有自己要忙的事，轨迹也基本定型。或已经积累足够的实习经验入职校招，或已经在父辈荫护下有长辈关照，或已经准备创业，或已经靠着不错的家底脱离竞争享受生活。

而我，只能像只无头苍蝇，在 BOSS，智联等 APP 消息列表中，看到一条条已读不回，心里无力感愈发强烈。

人活着，总是要有一些盼头？就比如我考研的那段日子，晨星带月，每早眼睛一睁，哦，我的任务是提高分数，多做几道题，多背几本书。短暂调整，直奔图书馆，又是忙碌而充实的一天，这也是我活下去的动力。

而我，自此之后，一蹶不振。考研彻底失败，春招也没有。每早醒来也只是望着天花板发呆很久，脑子空空的，没有任何动力，如行尸走肉一般。当下根本不知道该干什么，我又该何去何从？

考研失败，错过校招，再加上那段时间感情不顺，多重打击接连袭来，眉眼中多了些疲惫和倦怠。

那也是我此生中，或许是目前遇到的最大的失利，似乎人生跌至谷底。

伤心之余，日子还是要继续。某天，我释然了，也畅怀了。当下又还能如何呢？

或许是因为应激，或许是心有不甘，也或许是身上学生的稚嫩稚气未褪，不想过早投入社会。心中的想法愈发坚定，萌发生根：我要二战。

我的父母也没有多说什么，没有催着我找工作，也没有指责、埋怨。而是用缄默表达了对我的默许和支持。

我重新打开淘宝，下单了新版 880，660 系列的真题。

重新背起被我闲置已久的书包，重新踏上去图书馆的小路。宿舍距离图书馆不远，这条小路我走了三百多天，然而不过是两点一线、目不窥园。而我第一次见识陌上的冬去春来，云卷云舒时，才不由漾起一丝酸楚：我脱离世界太久了。

重逢阔别已久的图书馆，再次拿起手笔，看着之前书上做了无数遍的题，在草稿纸上慢慢演算既熟悉、又陌生的数学公式。

![](https://image-assets.dreams.plus/20250507141047173.jpg)

这次，我特意选了楼层高一点的自习室、靠窗的位置。

做题累了，就在座位上托腮远望深山、遍地绿树，看西北的风抚摸天上的云彩。月夜星辉时，我还特意盯着天上的月亮盈亏变化，正好数满一个月。

我的学校在山区，黄土高原，四周绿山环绕，夏季远处一览无余，晚上，没有城市的纸醉金迷的霓虹灯，能看见天上银河星光璀璨。

有时午后在座位上酣睡，高层遮挡较少，以阳光为被、以书本为枕。在日光沐浴下沉沉睡去。这样的日子，又持续了两个月。

![](https://image-assets.dreams.plus/20250507134057481.jpg)

早春过的很快，入夏，绿叶取代了招展的花枝，知了蝉鸣取代了春鸟嘤鸣。

我的本科迎来了尾声。毕业设计答辩完毕，同学们最后一次在校园内齐聚。

当我穿上学士袍、戴上学士帽，手捧毕业证书，恭恭敬敬接受院长挂穗，站在台上拍照时，我的本科四年结束了。

一声毕业照的快门咔嚓声响，给我四年的大学时间画上了句号。

![](https://image-assets.dreams.plus/20250507134551554.jpg)

但，我的脚步仍然没有结束。

身边的同学、室友一个接一个离开，打包行李，踏上轰鸣的火车，奔向远处，不知何时再相见。

而我，选择了留守学校，在图书馆，继续弥补我的遗憾。考研，是一个人的修行。

> 爱你来自于蛮荒，一生不借谁的光  \
> 你将造你的城邦，在废墟之上

## 生活还在继续

与图书馆为伴的时间很快，从五月份，就来到了七月份，作为毕业生，学校给我下达了最后通牒，毕业生必须限期离校。

我因为选择了二战，留守学校备考，此时也一时没了主意。

不过好在有人给我推荐了学校的「第二学士学位」项目，允许我在学校辅修一个双学位，到时候下学期可以继续留在学校，借用图书馆，心想也正合适我，于是申请了，被学校的第二学士学位录取。

九月，开学后，我再一次来到这个城市，这所学校。看见新生们在迎新大巴旁边叽叽喳喳，眼里尽是期待。

这个场景，忽然让我回到四年前，秋风习习，叶落梭梭，迎着朝霞。也是同一时间，我的父母亲自带着我给我送行，来到这所大学。

思绪回到从前，也是在 2020 年 9 月，我的父母带着我，坐了十几个小时的火车，三人满心欢喜踏入这个从未来过的西北重镇城市，来到这所学校里。

> 昔我往矣，杨柳依依  \
> 今我来思，雨雪霏霏

下火车时，正是凌晨五点，天刚刚破晓，黛紫染红了天空。

![](https://image-assets.dreams.plus/20250507133811658.jpg)

心中多了些悲凉。而今是我独自一人再次归来，继续我未完成的修行，是什么心情？满心悲凉、还是考研失败后的屈辱，或是对未来的迷茫。

再一次来到学校，校园里再也没有以前朝夕相处的同学，他们早已远去。

这下，在学校里，我就是彻底的孑然一身了。

入学后，我也认识了其他同样报了二学位的同学，不乏去年考研失败的、考公被刷的，跟我一样，都是立志于在校内再战，志在必得，继续未完成的事业。

老师允许我不上课，全心全力在图书馆备考。

我也如此。下一学期，我又开始投入紧张的备考，重拾去年时的状态。

在图书馆，继续刷我做了很多遍的题，回归到固定的轨迹，无非是四科。白天泡图书馆，晚上回到宿舍继续背单词、背书。这样的日子又持续了四个月。

2024 年的考研日期是 12 月 20 号左右，正值冬至，一年中黑夜最长的时候。这天我早早起来，冒着寒霜和黑夜踏入考场。

考试很快，下笔做题，就像我平常在图书馆里奋笔疾书写卷子那样，一气呵成，两天，做梦似的，就过去了。

## 落幕

4 月 3 日，早晨的闹钟把我唤醒。微信学信网给我弹出了一条通知，名单出来了。

悬着的心终于放下了，分数还可以，对得起我去年的努力。

考上了，拟录取通知下来了，看着报考系统上红色的「拟录取」时，刚开始或许有一点欣喜，但是没有想象的那么轰轰烈烈，并没有想象的那么开心。

考上了，然后，没然后了。

我想象中的，只要我接收到录取通知的时刻一到，我马上要买一箱啤酒，烤几个腰子牛排，暴饮暴食，熬夜通宵打游戏，想打多久就打多久，并没有发生。

只是蜷缩在昏暗的被窝里睡了很久很久。剩下的就是发呆，无尽无止地发呆、睡觉，过着混吃等死的日子。

偶尔也来到图书馆里，重温伏在书桌上的感觉。

![](https://image-assets.dreams.plus/20250507133510044.png)

> 草堂春睡足，窗外日迟迟

## 人生是旷野

打开很久没看的贴吧，信马由缰地浏览研究生学校的帖子。期间很幸运也找到了同学院的师兄，加上了他们的微信，向他们咨询了导师情况，接着跟师兄师姐聊了很晚，聊了读研生活和感想。

阴云很快再一次笼罩在脸上。

看着他们在抱怨读研压力大，一周两次组会，通宵做实验，甚至住在实验室，为论文数据发愁。

在四月中旬，我抽空给心仪的导师发了邮件，线下面谈选入组的事宜。

当我再次来到研究生学院，参观了实验室、工作室，逛了一圈，当我看到有师兄摆着一架行军床，上面还有要换的衣服和被子，我就知道这事肯定没那么简单。

是啊，我才理解到网上所说的，「研究生只有在录取和毕业的时候才是开心的」。

让我慢慢对研究生袪魅的是，读了研究生，也只不过是从一个牛马转换到另一个牛马而已，并不会让失业、收入改善多少。资本主义下任何人都只是待价而沽，出卖劳动力的工具人。决定就业的只是市场供需关系。

这也让我对研究生的生存逻辑有了更深入的了解。

就看现在的研究生「毕业即失业」现象比本科生并没有改善多少甚至还有严重趋势，说实话哈，有点不想读了。

**唉。**

当我们耗尽心力攀爬一座高山时，往往以为山顶的风景能消解所有疲惫，却可能发现，山顶的云雾遮蔽了预期中的壮丽。

这种"成功后的虚无感"恰如坦塔洛斯的处境，最渴求之物近在咫尺时，反而最令人清醒地认识到生命的本质。

当初我花费了那么多时间精力去考研，受了很多委屈，整个 2024 年都投入在上面，如今看来，这段经历是否值得？

人人都没有历史视角，我也不知道。但它就是个经历，是已经刻入我生命年表中的记录。

人生是旷野。以后的事情谁能说得清楚呢？未来的事情还是交给未来回答。至于读研是否值得，就当是投资未来了。

作为 00 后，我们所生活、经历的时代背景是十分魔幻和割裂的，这十几年恍如隔世，从网上调侃的「生于非典、考于新冠」，我经历了高考延期、三年封控、宿舍网课，也见识了多轮时代加速、急刹车等魔幻操作。

三年可以改变很多事，我是亲身经历过从 19 年到 23 年的变化的，18-19 年的时候社会的乐观顶峰，到 22 年又何其绝望与悲观。

「流动的现代性」在此显现得淋漓尽致。只能期望下一个三年好转吧。

当传统的人生脚本（本科、读研、工作、成家）越来越难以兑现承诺时，更需要培养冲浪者的生存智慧。就像在考研间隙观察月相变化的顿悟时刻，真正的救赎或许在于保持对生活诗意的凝视。

> 夫人之相与，俯仰一世，或取诸怀抱，悟言一室之内；或因寄所托，放浪形骸之外。虽趣舍万殊，静躁不同，当其欣于所遇，暂得于己，怏然自足，不知老之将至；及其所之既倦，情随事迁，感慨系之矣。向之所欣，俯仰之间，以为陈迹，犹不能不以之兴怀；况修短随化，终期于尽。古人云：“死生亦大矣。”岂不痛哉！

对我而言，这次考研二战的终极意义，或许不在改变外在境遇，而在培育一颗能感知四时流转、星移物换的丰盈心灵而已。


2025 年研招已经落幕，考上了，但似乎也并没有那么开心。日后，救赎之道仍待探索，毕竟，人生是旷野嘛。

考上了，但似乎也没有那么开心


在编写一些重要文档时需要插入一些示意图，如统计图、UML、流程图等，经常遇到 Markdown 图片和文本分离的问题，在 Markdown 中管理图片是比较麻烦的事，尤其是像统计图、UML 等需要频繁更新、版本管理的图片。

对于这类问题，我推荐的方案是 TTD (text-to-diagram). 现在有很多开源的免费的 TTD Generator，如 [Mermaid.js](https://mermaid.js.org/intro/)、[PlantUML](https://plantuml.com/en) 和 [GraphViz](https://graphviz.org/). 它们允许你使用文本形式的 DSL 代码去绘制图形，一般来说一份代码对应一种图像，非常利于文档中的插图。 [你可以在这里去体验它们](https://text-to-diagram.com/).

在这里我主要讨论 PlantUML、Graphviz 两种插件以及对应的不同情景。

## PlantUML

PlantUML 是老牌的、工业级的 UML 绘图工具，提供的 UML 也是最全的，很多教科书、学术论文等专业场景都采用的 PlantUML。因为“大而全”，所以它的学习曲线较为陡峭，文档内容也非常多，需要仔细研读。

[这是 PlantUML 的在线体验地址](https://www.plantuml.com/plantuml/)，以及[文档地址](https://plantuml.com/)。

我们写一个简单的图书馆系统的用例图，输入以下 PlantUML 代码：

```plaintext
@startuml
left to right direction

actor 读者 as Reader
actor 图书管理员 as Librarian

rectangle 图书馆管理系统 {
  usecase "搜索图书" as UC1
  usecase "借阅图书" as UC2
  usecase "归还图书" as UC3
  usecase "管理图书信息" as UC4
  usecase "生成借阅报告" as UC5

  Reader --> UC1
  Reader --> UC2
  Reader --> UC3

  Librarian --> UC4
  Librarian --> UC5

  UC2 .> UC1 : <<include>>  // 借阅前需搜索
  UC3 .> UC1 : <<include>>  // 归还前需搜索
  UC5 .> UC2 : <<extend>>   // 报告可扩展借阅细节
}
@enduml
```

它生成的用例图如下：

```plantuml
@startuml
left to right direction
actor 读者 as Reader
actor 图书管理员 as Librarian
rectangle 图书馆管理系统 {
  usecase "搜索图书" as UC1
  usecase "借阅图书" as UC2
  usecase "归还图书" as UC3
  usecase "管理图书信息" as UC4
  usecase "生成借阅报告" as UC5

  Reader --> UC1
  Reader --> UC2
  Reader --> UC3

  Librarian --> UC4
  Librarian --> UC5

  UC2 .> UC1 : <<include>>  // 借阅前需搜索
  UC3 .> UC1 : <<include>>  // 归还前需搜索
  UC5 .> UC2 : <<extend>>   // 报告可扩展借阅细节
}
@enduml
```

它生成的图像 UI 都有上个世纪的风格，比较适合对样貌美观性不高的场景，如教科书、论文等，好在可以自己选择主题。

但是 PlantUML 是基于 Java 编写的，运行它需要提供 Java 运行环境并放在后端中运行。但官方也把它打包成了一个 **图像生成的 Web 服务器**，把它部署在自己的服务器上就可以了。

使用 PlantUML Web 服务器的方法很简单，你只需要把要生成的 PlantUML 代码压缩、编码成 URL 参数，再传递给服务器即可。

就比如上面的 PlantUML，把代码编码，生成

```
VPBBIiD058RtWRp3eVjQ.....
```

然后就可以生成引用图片的 URL 了

```
![](https://www.plantuml.com/plantuml/VPBBIiD058RtWRp3eVjQ.....)
```

### VSCode 集成

PlantUML 有[官方 VSCode 插件](https://marketplace.visualstudio.com/items?itemName=jebbs.plantuml)，可以添加对其相关代码和语言的支持，当然也可以在 Markdown 预览视图中添加图像预览支持，只需要在 markdown 中编写代码，在预览中自动生成图像，就像这样。

![](https://github.com/qjebbs/vscode-plantuml/raw/master/images/markdown_demo.png)

### Unified.js 集成

如果需要和开源文档处理器 unified.js 集成，我编写了一个 remark 插件 [remark-refer-plantuml](https://github.com/PrinOrange/remark-refer-plantuml)，可以在 Markdown 文档语法树中自动完成识别、编码并替换成图像 URL 引用的这一过程。

## Graphviz

[Graphviz](https://gitlab.com/graphviz/graphviz) 是一个基础插图生成库，使用 DSL 来定义、生成图像，用算法自动排列节点和边，特别适合生成复杂的图表。

比如，用 Graphviz 写一个红黑树：

```plaintext
digraph RedBlackTree {
    node [shape=circle, style=filled, fontcolor=white];

    10 [fillcolor=black];
    5 [fillcolor=red];
    15 [fillcolor=red];
    3 [fillcolor=black];
    7 [fillcolor=black];
    12 [fillcolor=black];
    18 [fillcolor=black];

    10 -> 5;
    10 -> 15;
    5 -> 3;
    5 -> 7;
    15 -> 12;
    15 -> 18;
}
```

渲染效果图如下：

```graphviz-dot
digraph RedBlackTree {
    node [shape=circle, style=filled, fontcolor=white];

    10 [fillcolor=black];
    5 [fillcolor=red];
    15 [fillcolor=red];
    3 [fillcolor=black];
    7 [fillcolor=black];
    12 [fillcolor=black];
    18 [fillcolor=black];
    
    10 -> 5;
    10 -> 15;
    5 -> 3;
    5 -> 7;
    15 -> 12;
    15 -> 18;
}
```

或者写一个归并排序分治算法的流程示意图

```dot
digraph MergeSort {
    node [shape=plaintext];

    // 初始数组
    array [label="4  2  5  1  3", shape=box];

    // 递归拆分过程
    split1 [label="4  2  5   |   1  3", shape=box];
    split2_1 [label="4  2  |  5", shape=box];
    split2_2 [label="1  |  3", shape=box];
    split3_1 [label="4  |  2", shape=box];

    // 归并过程
    merge1_1 [label="2  4", shape=box];
    merge1_2 [label="1  3", shape=box];
    merge2 [label="2  4  5", shape=box];
    merge3 [label="1  2  3  4  5", shape=box];

    // 连接拆分过程
    array -> split1;
    split1 -> split2_1;
    split1 -> split2_2;
    split2_1 -> split3_1;
    split2_1 -> "5";
    split2_2 -> "1";
    split2_2 -> "3";
    split3_1 -> "4";
    split3_1 -> "2";

    // 连接合并过程
    "4" -> merge1_1;
    "2" -> merge1_1;
    "1" -> merge1_2;
    "3" -> merge1_2;
    merge1_1 -> merge2;
    "5" -> merge2;
    merge1_2 -> merge3;
    merge2 -> merge3;
}
```

Graphviz 的绘图功能也十分强大，根据代码生成 SVG 图像，但由于是底层库，对样式、布局算法等细节还需要自己手动完成，如果有自定义需求，通常需要二次开发。

Graphviz 是使用 C、C++ 语言进行开发的。也可以编译成 WASM，允许你直接在浏览器前端、Node.js 中使用。WASM 包有 [Viz.js](https://github.com/mdaines/viz-js)、[hpcc-js-wasm](https://github.com/hpcc-systems/hpcc-js-wasm) 等。

### VSCode 集成

VSCode 有对 Graphviz 语言的插件。不过，对在 VSCode Markdown 中预览视图的支持的插件不多，且缺乏维护。

所以我写了一个 VSCode 插件：[Markdown Graphviz Preview](https://marketplace.visualstudio.com/items?itemName=PrinOrange.markdown-graphviz-preview).

![](https://image-assets.dreams.plus/images202503031335435.png)

### Unified.js 集成

同样，我也写了一个 rehype 插件 [rehype-graphviz-diagram](https://www.npmjs.com/package/rehype-graphviz-diagram)，可以与 unified.js 文档处理集成。


在 Markdown 工程写作中，经常会有插入 UML，流程图，架构图等示意图的需求。本文推荐两种工具以及对应的 Markdown 处理库、Visual Studio Code Markdown 扩展插件。

工程写作：Markdown 插入示意图


I have tried to do technical writing in English on intend since 2022, when I made my mind of putting wheels on my shoulder for English study. Actually I have tried my English writing skill on intend by this way for make my language level better improved. And during these years I have experienced many challenges and troubles in conquering English.

The first one is vocabulary size. I allocated many of my vigors on expanding vocabularies and accumulated English expression particles in studying and reading. But soon I realized that reading technical text and literature text are two different realm logic.

The technical text are legible with perspicuous grammar syntax structure, and that's why I could read many of opensource project documents without any difficulties. The only challenge is reserve of terms and jargons. That's why do we need specialized professional English, for English has been contaminated by more loan words, and there are more amorphous word ideographic stemmer rules in each terms. To be honest I have complain about English more on this problem, it erects human-made thresholds among variable industries. The topic seems too be far-fetched. Let's go back to the technical writing. The another problem is English term are highly specialized, that is to say polysemy may lead to ambiguity if without any context details. For example, the word *layer* has distinguish meanings in software development and stratum geology. Also non-native speakers tend to confuse synonyms (e.g., "accuracy" vs. "precision") and need to rely on terminology databases or industry standards (e.g., IEEE Standard Glossary).

For special target portrayed, the passive voice in English is frequently used in technical writing (e.g. "The experiment was conducted..."). Non-native speakers may overuse the active voice, which affects objectivity. Complex sentences (e.g. conditional clauses, participle structures) are prone to grammatical errors. The concise principle of technical writing should be followed (e.g. IEEE recommends using short sentences).

Improper use of logical adjunction-connectors ("therefore", "however") between paragraphs can weaken the chain of reasoning. The English technical writing needs rigorousity suits for logical deduction chains. But for non-native speakers, the English adjunction connector words are too far exquisite to comprehend them easily.

Here, I found a pretty worth reading book titled *Science Research Writing for Non-native speakers of English*. [Here is the download link](https://z-library.sk/book/25451495/4d36ea/science-research-writing-for-native-and-nonnative-speakers-of-english.html). It's a practical guide designed to help researchers and students whose first language is not English master the conventions of scientific writing in English. Authored by Hilary Glasman-Deal, the book focuses on the structure, style, and clarity required in academic research papers. It systematically breaks down each section of a scientific manuscript—Introduction, Methods, Results, Discussion (IMRaD), and Abstract—providing clear templates, annotated examples, and phrase-level guidance.

Here is the table of contents:

![](https://image-assets.dreams.plus/images202502122348927.png)

Based on problem mentioned above, I have made a plan for training technical writing in English. This 12-week plan is designed to systematically improve scientific writing skills for technical writing. Adjust intensity based on schedule.

**Phase 1: Foundations (Weeks 1–3)**  
**Goal**: Master structure, grammar basics, and discipline-specific vocabulary.  

1. **Reverse-Engineer Papers**  
   - Select 3 high-impact papers in your field (e.g., from *Nature* or your target journal).  
   - Annotate each IMRaD section: Highlight how hypotheses/results are framed, transitions between paragraphs, and verb tenses.  
   - Create a "phrase bank" of recurring patterns (e.g., "Our data suggest..." in Discussions).  

2. **Grammar & Style Drills**  
   - Daily: Complete 1–2 exercises from *Science Research Writing for Non-Native Speakers* (e.g., tense consistency in Methods).  
   - Use *Academic Phrasebank* (University of Manchester) to practice hedging/emphatic language.  
   - Tools: Enable "Technical Writing" mode in Grammarly; analyze suggestions for passive voice/precision.  

3. **Vocabulary Building**  
   - Build a glossary of 50+ field-specific terms using *Elsevier’s Dictionary of Technical Terms*.  
   - Practice synonym differentiation (e.g., "demonstrate" vs. "indicate") via quizzes on *Quizlet*.  

**Phase 2: Drafting & Precision (Weeks 4–6)**  
**Goal**: Write structured drafts with clear logic and technical accuracy.  

1. **Section-by-Section Practice**  
   - **Abstract**: Write 10 one-sentence summaries of existing papers, then expand to 250-word abstracts.  
   - **Methods**: Describe a lab procedure twice: once in active voice (blog style), once in passive voice (journal style).  
   - **Results**: Convert raw data (graphs/tables) into 3 versions: overly detailed, too vague, and Goldilocks-level.  

2. **Peer Simulation**  
   - Join *r/AskAcademia* or *ResearchGate* to share drafts for feedback.  
   - Use the "Three Layers" editing technique:  
     - Layer 1: Logic flow (Does each paragraph answer "So what?").  
     - Layer 2: Technical accuracy (Verify units, abbreviations, citations).  
     - Layer 3: Language (Prepositions/articles, conciseness).  

3. **Tools Deep Dive**  
   - Learn LaTeX basics (Overleaf tutorials) for equations/formatting.  
   - Run drafts through *Hemingway Editor* to eliminate "Very Hard to Read" sentences.  

**Ongoing Habits:**  

- **Weekly**: Critique 1 poorly written paper (e.g., retracted papers on *Retraction Watch*) to recognize red flags.  
- **Monthly**: Attend 1 virtual workshop (e.g., *ACS Webinars* on writing).  
- **Toolkit**: Maintain a "Writing Swipe File" (YuQue/Notion) for brilliant sentences/structures you encounter.  

Hope for good progress.


I have tried to do technical writing in English since 2022. And now I have came up with a further plan.

About Technical Writing in English.


I'm planning to do some optimizations for full-text search engine for my blog site recently and enhance it with better Chinese and English language support, and there are some ideas flips through my mind. For all, there are some hands-on service like [Algolia](https://www.algolia.com/), but I prefer do explorations on my own over this realm and now I have tried to conclude some tips.

Generally speaking, to devise a full-text search engine, it should be got through these following processions when I get a raw text.

- **Entering meta data**: The metadata, which is common for every texts but indicates their uniques, such as title, author, keywords and summary. This is not difficult and different programs have their own procession details.
- **Remove redundant ingredients**: Most of texts are formatted with markup syntax to express their roles in text components, which unnecessarily should not to be entered in index. For example, the `<img/>` element could be removed from text by library [JSDOM](https://github.com/jsdom/jsdom), [Unified](https://github.com/unifiedjs/unified) and so on.
- **Remove markups**: The markup syntax that express their roles in text components also could be removed before indexing text. Like HTML tags and markdown markup syntax also could be removed by JSDOM or Unified.js.
- **Tokenization**: Is a procession of great significance during the indexing, also what's this article mainly talking about. It divide a sentence or paragraph into words by special grammar rules. In most cases, the keywords should be more related to main ideas of the sentence or paragraph it belonged to. A good tokenization work are benefit to more accuracy for searching relations. But let just tokenization itself alone is not enough for work, sometimes it needs to be further processed, as part-of-speech tagging, removing stopwords and extracting word stems.
- **Indexing into data structure**: Map the word set to its corresponding article and establish mapping relations with specified data structure(Inverted Index, Hash structure and Tire tree, etc), and ranking them by weight coefficient with specified rules from algorithm.

## Getting Start with English

The English tokenizer is relatively easy to devise in comparison with Chinese and Japanese for words in English sentence are separated by blanks naturally. It's no need to tokenize words like Chinese sentence with grammar structure analyzing. We could simply make a tokenizer with JavaScript for given English sentence.

```javascript
console.log("A quick fox jumps over the lazy dog.".spilt(" "));
// Output: ["a","quick","fox","jumps","over","the","lazy","dog"]
```

But that's not enough. For the sentence "I'm hungry" is combined with three words actually. The " I'm " inside the sentence in fact is abbreviation combined with "I" and "am", it should be separated into "I", "am", "hungry" rather than "I'm" and "hungry". The same situation includes "haven't", "didn't" and so on. But this situation is easy to do. You can make a replacement before word tokenization.

```javascript
function expandContractions(sentence) {
  const contractionsMap = {
    "I'm": "I am",
    "you're": "you are",
    "he's": "he is",
    "she's": "she is",
    "it's": "it is",
    "we're": "we are",
    "they're": "they are",
    // More rules here...
    "what's": "what is",
  };
  const regex = new RegExp(
    "\\b(" + Object.keys(contractionsMap).join("|") + ")\\b",
    "gi"
  );
  return sentence.replace(regex, (match) => contractionsMap[match]);
}

const sentence =
  "I'm learning JavaScript and it's fun! He didn't know you were coming.";
const expandedSentence = expandContractions(sentence);
console.log(expandedSentence.spilt(" "));
```

That works well. But another problem arises: English is a inflectional language with many functional words in corpus, like "am", "where", "that" and "on", which contributes many proportions but have less actual meanings in language context, we also name them as "stopword". Although they have little practical significance because of their high frequency of occurrence, they contribute significantly to the indexing of documents, thus reducing the relevance of search results.

Especially in TF-IDF Statistical Model, the stops are of large frequency and its IDF approaches to zero with limited contributions, therefore, removing stop words will not significantly affect the calculation results.

These could be safely removed from indexing.

There are quite a few opensource stopword library in English on npm and github, so you can use them on hand naturally. There is an open source project [stopword](https://github.com/fergiemcdowall/stopword) that contains stop words for many languages. Let's take the `stopword` package as an example. [Its documentation can be found here](https://github.com/fergiemcdowall/stopword?tab=readme-ov-file#usage).

```bash
npm install stopword
```

Then use this library to remove stop words to achieve the effect of streamlined word tokenization.

```javascript
import { removeStopwords } from "stopword";
const oldString = "a really Interesting string with some words".split(" ");
const newString = removeStopwords(oldString);
// newString is now [ 'really', 'Interesting', 'string', 'words' ]
```

This will prevent unnecessary stop words from taking up a lot of indexing progress. However, this is still not enough.

We noticed these following two sentence,

> She decided to paint the wall blue. \
> She made a decision to paint the wall blue.

For these two sentences the "decide" is verb while the "decision" is a noun, and they express the same thing in two different grammar structures. Examples abound in English, such as "fond" and "fondness", "disagree" and "disagreement", etc.

But if you tokenize it on autopilot simply and crudely with no optimization, the index will perceive `decide` and `decision` are two irrelevant words. And it will not offer the result "... decision to paint ..." if you search "... decide to paint ...". Therefore what's next to do is make index perceive `decision` and `decide` as one word so as to improve the relation of searching.

Indo-European languages ​​have their own set of word formation. The same is true for English.

Generally speaking, words are composed of stems and affixes. The stems are used to express the original meaning, and an initial word with a suffix becomes a derived word. There are suffixes such as `-ing`, `-tion`, `-able`, `-ism`, etc., which are used to express part of speech and auxiliary meaning.

So, what's the common between the words `decision` and `decide`? Yes, they have the same word stem `decid`, likewise it also includes `decisive`,`decided`,`deciding`,`decides`,`decider`, etc. They could be perceived as the same word if they have same word stem.

There is a package to extract stem from word by linguistical theory for English words named [stemmer](https://www.npmjs.com/package/stemmer).

```bash
npm install stemmer
```

To extract word stem from a given word,

```javascript
import { stemmer } from "stemmer";

stemmer("considerations"); // => 'consider'
stemmer("detestable"); // => 'detest'
stemmer("vileness"); // => 'vile'
```

The problem of stems has been solved now, and it can further transform all different forms of words into a unified word stem to optimizing the indexing effect.

Think about how else you can optimize search relevance further?

We notice that

> She purchases some discounted goods after work. \
> She buys some discounted goods after work.

Here the word `Purchase` and `Buy` are synonyms to each other and these two sentence use two different word but express one same meaning.

If the indexer only includes the sentence 'She purchases some discounted goods after work' , then searching for 'buy some goods' would similarly fail to retrieve any relevant results.

So we can do optimizations in aspect of synonym thesaurus.

Let's contrive such a work, to tokenize a sentence it not only outputs the words which exist in this sentence but also outputs their synonyms, for example, the sentence

```plaintext
She purchases some discounted goods after work.
```

will outputs such result

```plaintext
She, her, herself
purchases, buys, acquires, gets, obtains, procures
some, a few, several, certain, various
discounted, reduced, sale, cheaper, marked down, cut-price
goods, merchandise, products, items, commodities, wares
after, following, post-, subsequent to, behind
work, job, employment, labor, task, occupation
```

Then if I search "She procures a few commodities after labors." the origin sentence also is going to be provided and prompted, that's to enlarge the searching ranges and result spaces.

> **NOTE:** Strictly there are no absolute "two words of equivalence" in English language. Every two words of synonym you meet always have their blur and subtle nuance in context.

We might as well add synonyms to the result while tokenization. And there are also libraries on npm that allows you query synonyms for correspond English word, and the common one is `natural`.

```bash
npm i natural
```

And outputs the tokenization

```typescript
import natural from "natural";
import { stemmer } from "stemmer";
const wordnet = new natural.WordNet();

async function tokenize(sentence: string): Promise<string[]> {
  const tokenizer = new natural.WordTokenizer();
  const tokens = tokenizer.tokenize(sentence);
  const resultsSet = new Set<string>();
  tokens.forEach((token) => resultsSet.add(stemmer(token)));
  const lookupPromises = tokens.map(
    (token) =>
      new Promise<void>((resolve) => {
        wordnet.lookup(token, (definitions) => {
          definitions.forEach((definition) => {
            definition.synonyms.forEach((synonym) => {
              const stemmedSynonym = stemmer(synonym);
              resultsSet.add(stemmedSynonym);
            });
          });
          resolve();
        });
      })
  );
  await Promise.all(lookupPromises);
  return Array.from(resultsSet);
}
```

This code will not only tokenize a sentence into words but also their synonyms as a result which will highly improved the range of search accuracy.

## Support Chinese

Chinese sentence has no blanks to separate every words compared to English one, and tokenization work on Chinese sentence could only be relayed on grammatical rules with linguistical analysis, which also causes more extra costs than English. So there are something Chinese tokenization models based on Artificial Intelligence and most of them are paid. Fortunately there is one powerful and free tokenizer named [Jieba](https://github.com/fxsjy/jieba) maintained by opensource community, and it satisfies most of out needs just for personal blog indexing. Here is a correspond [node.js binding library](https://www.npmjs.com/package/@node-rs/jieba).

Secondly, Chinese is not inflected but isolating languages that means Chinese word has no tense, sexual and personal changes, that is to say we need not take them into considerations when processing a word like extracting word stems. And other remain processing are same to English, removing stopwords, adding synonyms.

But most of blogs written in Chinese, especially technical writings, they always mixed with some English terms and sentences. So the sporadic English words exist in Chinese sentence should be considered and processed either.

This is actually very easy to do. The Jieba could separate not only Chinese words but also English words mixed in sentence, and we distinguish Chinese words and English words from separation result, and process English words by the measures above mentioned, and process Chinese and English words respectively.

```typescript
import { Jieba } from "@node-rs/jieba";
import stopword from "stopword"; 

export function ChineseLanguageTokenizer(str: string): string[] {
  const NonChineseAndLatinRegex =
    /[^\u4e00-\u9fff\u0041-\u005a\u0061-\u007a\u00c0-\u017f\u0180-\u024f\s]/g;
  const jieba = new Jieba();
  const cleanedStr = str.replace(NonChineseAndLatinRegex, " ");

  const tokens = jieba.cutForSearch(cleanedStr, true);

  const englishWords = stopword.removeStopwords(
    tokens.filter((word) => /^[a-zA-Z]+$/.test(word))
  );
  const processedEnglishWords = englishWords.map(addSynonyms).map(extractStem);

  const chineseWords = stopword.removeStopwords(
    tokens.filter((word) => /^[\u4e00-\u9fff]+$/.test(word)),
    stopword.zho
  ).map(addSynonyms);

  const result = [...chineseWords, ...processedEnglishWords];

  return result;
}
```

By this way we devise a tokenizer supports Chinese-English mixed text, it works well on all English text, all Chinese text, and Chinese texts mixed with a little English terms.

## Japanese

Compared to English and Chinese, **it's far troublesome to devise a Japanese Tokenizer** for Japanese language feature of written system. Because Japanese text has two written forms: Kanji (漢字) and Kana (假名) so that many Japanese words could be written in both Kanji and Kana.

However, the Japanese syllable is very simple, and usually one kana corresponds to one syllable, so homonym situations are very serious. Therefore, in Japanese written texts, some Japanese words often need to be written as Chinese characters to distinguish them.

For example, these words have same Kana form (かん) but different meanings.

- 缶 (かん) - means the Jars.
- 観 (かん) - means "watching" and "observing".
- 漢 (かん) - means "Han nationality" or "Chinese Character".
- 幹 (かん) - means "cadre" or "mainstay".

This homographs situation sometimes will make confusion and comprehending mess. For example, here is a Japanese sentence「おはしをください」。In this sentence **「はし」** is a homograph with three irrelevant meanings.

1. 箸（はし）: Chopstick.
2. 橋（はし）: Bridge.
3. 端（はし）: Edge, end of something.

From the perspective of grammar and common sense, this sentence is completely valid. However, without a specific context, it is impossible to determine what the speaker is requesting.

- **If the context is a restaurant**, then「はし」could be **chopsticks**, and the sentence would mean "Please give me chopsticks".
- **If the context is asking for directions**, then「はし」could be **bridge**, and the sentence would mean "Please tell me where the bridge is" or "Please take me to the bridge."
- **If the context is a physical location**, such as the edge of a table, then「はし」could be **edge**, and the sentence would mean "Please pass something to the edge".

Only when with the specific context can the true meaning of "はし" in this sentence be determined. Judging from the grammar and literal meaning alone, all three meanings are valid and unrelated.

In my point of view, the mere tokenization itself is not true of Japanese. Japanese should be tokenized by "ideas" rather than "words".

Take the sentence mentioned above for example, if the context is in a restaurant,

- お：A linker that expresses respect or politeness and is used to modify the following noun.
- はし：A noun with meanings of "chopstick".
- を：A particle that marks the object.
- ください：The imperative form of the verb "下さい", used to express a request, meaning "please".

And it outputs a set of ideas `Chopstick`, `Please` as tokenization result. Fortunately, the Kanji in Japanese has its function for marking the ideas so we can perceive kanas as one same word if they have same Kanji form.

So I take these steps for Japanese Tokenization:

1. Separate sentences into components and groups of words by grammar rules.
2. Remove functional words and stopwords.
3. Revert every words to basic forms.
4. Convert Kana into ideographic kanji as much as possible, and outputs the ideas set as result.

The processing of synonyms, and Japanese-English mixed writing can also refer to the steps mentioned above, they are similar by and large.

The common Japanese tokenizer in community are [Mecab](https://taku910.github.io/mecab/), [Sudachi](https://github.com/WorksApplications/Sudachi) and [Kuromoji](https://www.atilika.org/). But I have sift them through but they are outmode and in lack of maintain for years...

Just make do with them... if you do not have better alternative... (I want to complain the technical atmosphere in Japan by the way...It's hard to put my comments on Japanese opensource community...)

## Epilogue

Of course, the tokenizer designed above is just a very simple implementation. Although it is very simple, it has all the necessary effects. However, it is sufficient for designing of search engine for personal blog. More advanced full-text search indexing features such as semantic deduction belong to the research realm of NLP in artificial intelligence. They have high requirements on device performance and training data. If it is a small site, it is really unnecessary for cost considerations.


Take Chinese, English and Japanese for Example

I'm planning to do some optimizations for full-text search engine for my blog site recently and enhance it with better Chinese and English language support, and there are some ideas flips through my mind.

MY BLOG