
I'm planning to do some optimizations for full-text search engine for my blog site recently and enhance it with better Chinese and English language support, and there are some ideas flips through my mind. For all, there are some hands-on service like [Algolia](https://www.algolia.com/), but I prefer do explorations on my own over this realm and now I have tried to conclude some tips.

Generally speaking, to devise a full-text search engine, it should be got through these following processions when I get a raw text.

- **Entering meta data**: The metadata, which is common for every texts but indicates their uniques, such as title, author, keywords and summary. This is not difficult and different programs have their own procession details.
- **Remove redundant ingredients**: Most of texts are formatted with markup syntax to express their roles in text components, which unnecessarily should not to be entered in index. For example, the `<img/>` element could be removed from text by library [JSDOM](https://github.com/jsdom/jsdom), [Unified](https://github.com/unifiedjs/unified) and so on.
- **Remove markups**: The markup syntax that express their roles in text components also could be removed before indexing text. Like HTML tags and markdown markup syntax also could be removed by JSDOM or Unified.js.
- **Tokenization**: Is a procession of great significance during the indexing, also what's this article mainly talking about. It divide a sentence or paragraph into words by special grammar rules. In most cases, the keywords should be more related to main ideas of the sentence or paragraph it belonged to. A good tokenization work are benefit to more accuracy for searching relations. But let just tokenization itself alone is not enough for work, sometimes it needs to be further processed, as part-of-speech tagging, removing stopwords and extracting word stems.
- **Indexing into data structure**: Map the word set to its corresponding article and establish mapping relations with specified data structure(Inverted Index, Hash structure and Tire tree, etc), and ranking them by weight coefficient with specified rules from algorithm.

## Getting Start with English

The English tokenizer is relatively easy to devise in comparison with Chinese and Japanese for words in English sentence are separated by blanks naturally. It's no need to tokenize words like Chinese sentence with grammar structure analyzing. We could simply make a tokenizer with JavaScript for given English sentence.

```javascript
console.log("A quick fox jumps over the lazy dog.".spilt(" "));
// Output: ["a","quick","fox","jumps","over","the","lazy","dog"]
```

But that's not enough. For the sentence "I'm hungry" is combined with three words actually. The " I'm " inside the sentence in fact is abbreviation combined with "I" and "am", it should be separated into "I", "am", "hungry" rather than "I'm" and "hungry". The same situation includes "haven't", "didn't" and so on. But this situation is easy to do. You can make a replacement before word tokenization.

```javascript
function expandContractions(sentence) {
  const contractionsMap = {
    "I'm": "I am",
    "you're": "you are",
    "he's": "he is",
    "she's": "she is",
    "it's": "it is",
    "we're": "we are",
    "they're": "they are",
    // More rules here...
    "what's": "what is",
  };
  const regex = new RegExp(
    "\\b(" + Object.keys(contractionsMap).join("|") + ")\\b",
    "gi"
  );
  return sentence.replace(regex, (match) => contractionsMap[match]);
}

const sentence =
  "I'm learning JavaScript and it's fun! He didn't know you were coming.";
const expandedSentence = expandContractions(sentence);
console.log(expandedSentence.spilt(" "));
```

That works well. But another problem arises: English is a inflectional language with many functional words in corpus, like "am", "where", "that" and "on", which contributes many proportions but have less actual meanings in language context, we also name them as "stopword". Although they have little practical significance because of their high frequency of occurrence, they contribute significantly to the indexing of documents, thus reducing the relevance of search results.

Especially in TF-IDF Statistical Model, the stops are of large frequency and its IDF approaches to zero with limited contributions, therefore, removing stop words will not significantly affect the calculation results.

These could be safely removed from indexing.

There are quite a few opensource stopword library in English on npm and github, so you can use them on hand naturally. There is an open source project [stopword](https://github.com/fergiemcdowall/stopword) that contains stop words for many languages. Let's take the `stopword` package as an example. [Its documentation can be found here](https://github.com/fergiemcdowall/stopword?tab=readme-ov-file#usage).

```bash
npm install stopword
```

Then use this library to remove stop words to achieve the effect of streamlined word tokenization.

```javascript
import { removeStopwords } from "stopword";
const oldString = "a really Interesting string with some words".split(" ");
const newString = removeStopwords(oldString);
// newString is now [ 'really', 'Interesting', 'string', 'words' ]
```

This will prevent unnecessary stop words from taking up a lot of indexing progress. However, this is still not enough.

We noticed these following two sentence,

> She decided to paint the wall blue. \
> She made a decision to paint the wall blue.

For these two sentences the "decide" is verb while the "decision" is a noun, and they express the same thing in two different grammar structures. Examples abound in English, such as "fond" and "fondness", "disagree" and "disagreement", etc.

But if you tokenize it on autopilot simply and crudely with no optimization, the index will perceive `decide` and `decision` are two irrelevant words. And it will not offer the result "... decision to paint ..." if you search "... decide to paint ...". Therefore what's next to do is make index perceive `decision` and `decide` as one word so as to improve the relation of searching.

Indo-European languages ​​have their own set of word formation. The same is true for English.

Generally speaking, words are composed of stems and affixes. The stems are used to express the original meaning, and an initial word with a suffix becomes a derived word. There are suffixes such as `-ing`, `-tion`, `-able`, `-ism`, etc., which are used to express part of speech and auxiliary meaning.

So, what's the common between the words `decision` and `decide`? Yes, they have the same word stem `decid`, likewise it also includes `decisive`,`decided`,`deciding`,`decides`,`decider`, etc. They could be perceived as the same word if they have same word stem.

There is a package to extract stem from word by linguistical theory for English words named [stemmer](https://www.npmjs.com/package/stemmer).

```bash
npm install stemmer
```

To extract word stem from a given word,

```javascript
import { stemmer } from "stemmer";

stemmer("considerations"); // => 'consider'
stemmer("detestable"); // => 'detest'
stemmer("vileness"); // => 'vile'
```

The problem of stems has been solved now, and it can further transform all different forms of words into a unified word stem to optimizing the indexing effect.

Think about how else you can optimize search relevance further?

We notice that

> She purchases some discounted goods after work. \
> She buys some discounted goods after work.

Here the word `Purchase` and `Buy` are synonyms to each other and these two sentence use two different word but express one same meaning.

If the indexer only includes the sentence 'She purchases some discounted goods after work' , then searching for 'buy some goods' would similarly fail to retrieve any relevant results.

So we can do optimizations in aspect of synonym thesaurus.

Let's contrive such a work, to tokenize a sentence it not only outputs the words which exist in this sentence but also outputs their synonyms, for example, the sentence

```plaintext
She purchases some discounted goods after work.
```

will outputs such result

```plaintext
She, her, herself
purchases, buys, acquires, gets, obtains, procures
some, a few, several, certain, various
discounted, reduced, sale, cheaper, marked down, cut-price
goods, merchandise, products, items, commodities, wares
after, following, post-, subsequent to, behind
work, job, employment, labor, task, occupation
```

Then if I search "She procures a few commodities after labors." the origin sentence also is going to be provided and prompted, that's to enlarge the searching ranges and result spaces.

> **NOTE:** Strictly there are no absolute "two words of equivalence" in English language. Every two words of synonym you meet always have their blur and subtle nuance in context.

We might as well add synonyms to the result while tokenization. And there are also libraries on npm that allows you query synonyms for correspond English word, and the common one is `natural`.

```bash
npm i natural
```

And outputs the tokenization

```typescript
import natural from "natural";
import { stemmer } from "stemmer";
const wordnet = new natural.WordNet();

async function tokenize(sentence: string): Promise<string[]> {
  const tokenizer = new natural.WordTokenizer();
  const tokens = tokenizer.tokenize(sentence);
  const resultsSet = new Set<string>();
  tokens.forEach((token) => resultsSet.add(stemmer(token)));
  const lookupPromises = tokens.map(
    (token) =>
      new Promise<void>((resolve) => {
        wordnet.lookup(token, (definitions) => {
          definitions.forEach((definition) => {
            definition.synonyms.forEach((synonym) => {
              const stemmedSynonym = stemmer(synonym);
              resultsSet.add(stemmedSynonym);
            });
          });
          resolve();
        });
      })
  );
  await Promise.all(lookupPromises);
  return Array.from(resultsSet);
}
```

This code will not only tokenize a sentence into words but also their synonyms as a result which will highly improved the range of search accuracy.

## Support Chinese

Chinese sentence has no blanks to separate every words compared to English one, and tokenization work on Chinese sentence could only be relayed on grammatical rules with linguistical analysis, which also causes more extra costs than English. So there are something Chinese tokenization models based on Artificial Intelligence and most of them are paid. Fortunately there is one powerful and free tokenizer named [Jieba](https://github.com/fxsjy/jieba) maintained by opensource community, and it satisfies most of out needs just for personal blog indexing. Here is a correspond [node.js binding library](https://www.npmjs.com/package/@node-rs/jieba).

Secondly, Chinese is not inflected but isolating languages that means Chinese word has no tense, sexual and personal changes, that is to say we need not take them into considerations when processing a word like extracting word stems. And other remain processing are same to English, removing stopwords, adding synonyms.

But most of blogs written in Chinese, especially technical writings, they always mixed with some English terms and sentences. So the sporadic English words exist in Chinese sentence should be considered and processed either.

This is actually very easy to do. The Jieba could separate not only Chinese words but also English words mixed in sentence, and we distinguish Chinese words and English words from separation result, and process English words by the measures above mentioned, and process Chinese and English words respectively.

```typescript
import { Jieba } from "@node-rs/jieba";
import stopword from "stopword"; 

export function ChineseLanguageTokenizer(str: string): string[] {
  const NonChineseAndLatinRegex =
    /[^\u4e00-\u9fff\u0041-\u005a\u0061-\u007a\u00c0-\u017f\u0180-\u024f\s]/g;
  const jieba = new Jieba();
  const cleanedStr = str.replace(NonChineseAndLatinRegex, " ");

  const tokens = jieba.cutForSearch(cleanedStr, true);

  const englishWords = stopword.removeStopwords(
    tokens.filter((word) => /^[a-zA-Z]+$/.test(word))
  );
  const processedEnglishWords = englishWords.map(addSynonyms).map(extractStem);

  const chineseWords = stopword.removeStopwords(
    tokens.filter((word) => /^[\u4e00-\u9fff]+$/.test(word)),
    stopword.zho
  ).map(addSynonyms);

  const result = [...chineseWords, ...processedEnglishWords];

  return result;
}
```

By this way we devise a tokenizer supports Chinese-English mixed text, it works well on all English text, all Chinese text, and Chinese texts mixed with a little English terms.

## Japanese

Compared to English and Chinese, **it's far troublesome to devise a Japanese Tokenizer** for Japanese language feature of written system. Because Japanese text has two written forms: Kanji (漢字) and Kana (假名) so that many Japanese words could be written in both Kanji and Kana.

However, the Japanese syllable is very simple, and usually one kana corresponds to one syllable, so homonym situations are very serious. Therefore, in Japanese written texts, some Japanese words often need to be written as Chinese characters to distinguish them.

For example, these words have same Kana form (かん) but different meanings.

- 缶 (かん) - means the Jars.
- 観 (かん) - means "watching" and "observing".
- 漢 (かん) - means "Han nationality" or "Chinese Character".
- 幹 (かん) - means "cadre" or "mainstay".

This homographs situation sometimes will make confusion and comprehending mess. For example, here is a Japanese sentence「おはしをください」。In this sentence **「はし」** is a homograph with three irrelevant meanings.

1. 箸（はし）: Chopstick.
2. 橋（はし）: Bridge.
3. 端（はし）: Edge, end of something.

From the perspective of grammar and common sense, this sentence is completely valid. However, without a specific context, it is impossible to determine what the speaker is requesting.

- **If the context is a restaurant**, then「はし」could be **chopsticks**, and the sentence would mean "Please give me chopsticks".
- **If the context is asking for directions**, then「はし」could be **bridge**, and the sentence would mean "Please tell me where the bridge is" or "Please take me to the bridge."
- **If the context is a physical location**, such as the edge of a table, then「はし」could be **edge**, and the sentence would mean "Please pass something to the edge".

Only when with the specific context can the true meaning of "はし" in this sentence be determined. Judging from the grammar and literal meaning alone, all three meanings are valid and unrelated.

In my point of view, the mere tokenization itself is not true of Japanese. Japanese should be tokenized by "ideas" rather than "words".

Take the sentence mentioned above for example, if the context is in a restaurant,

- お：A linker that expresses respect or politeness and is used to modify the following noun.
- はし：A noun with meanings of "chopstick".
- を：A particle that marks the object.
- ください：The imperative form of the verb "下さい", used to express a request, meaning "please".

And it outputs a set of ideas `Chopstick`, `Please` as tokenization result. Fortunately, the Kanji in Japanese has its function for marking the ideas so we can perceive kanas as one same word if they have same Kanji form.

So I take these steps for Japanese Tokenization:

1. Separate sentences into components and groups of words by grammar rules.
2. Remove functional words and stopwords.
3. Revert every words to basic forms.
4. Convert Kana into ideographic kanji as much as possible, and outputs the ideas set as result.

The processing of synonyms, and Japanese-English mixed writing can also refer to the steps mentioned above, they are similar by and large.

The common Japanese tokenizer in community are [Mecab](https://taku910.github.io/mecab/), [Sudachi](https://github.com/WorksApplications/Sudachi) and [Kuromoji](https://www.atilika.org/). But I have sift them through but they are outmode and in lack of maintain for years...

Just make do with them... if you do not have better alternative... (I want to complain the technical atmosphere in Japan by the way...It's hard to put my comments on Japanese opensource community...)

## Epilogue

Of course, the tokenizer designed above is just a very simple implementation. Although it is very simple, it has all the necessary effects. However, it is sufficient for designing of search engine for personal blog. More advanced full-text search indexing features such as semantic deduction belong to the research realm of NLP in artificial intelligence. They have high requirements on device performance and training data. If it is a small site, it is really unnecessary for cost considerations.


Take Chinese, English and Japanese for Example

I'm planning to do some optimizations for full-text search engine for my blog site recently and enhance it with better Chinese and English language support, and there are some ideas flips through my mind.

Some Tips for Designing Tokenizer for Search Engine


Node.js adopts an event-driven and non-blocking I/O model, allowing it to efficiently handle a large number of I/O requests and concurrent connections, which is crucial for a streaming media server. Meanwhile, FFmpeg, as a powerful video processing tool, provides rich video processing capabilities.

Combining Node.js's powerful extension capabilities, we can easily integrate FFmpeg to create a feature-complete video streaming media server, meeting various streaming media application requirements.

Now let's start from scratch and implement a simple video server: implementing the functionality to upload and play videos. In the process of learning, we will gradually understand how the video server works.

Before we begin, you need to:

- Understand the basic API of Node.js, such as the fs module, process module, etc.
- Understand the usage and common APIs of Express.js.
- Install and understand the basic usage of FFMPEG.

Now we want to implement a streaming media server. There are many streaming media protocols and formats on the market (RTP/RTCP/RTSP/RTMP/MMS/HLS, etc.), each with different applications. However, the vast majority of websites currently use HLS/M3U8 format streaming media, which is also supported by modern browsers.

Therefore, this article does not elaborate on these streaming media protocols but implements a simple HLS/M3U8 format streaming media server.

**The streaming media server implemented in this article is not technically difficult.**

## MP4 Video Slicing

Open some mainstream video websites, switch to the "Network" tab in Chrome Devtools, and then play the video. You can see that the browser is constantly downloading small files from a URL. These small files are video segments.

In general, web videos do not directly download the entire video file because a video file is usually several gigabytes, and downloading the entire file would be too large. Instead, the videos are split into many small segments, which are then continuously transferred to the browser. The browser then sequentially plays these segments.

For HLS/M3U8 type streaming media, the video playback address is the address for downloading the m3u8 file. The m3u8 file is a text-based list that stores the names, addresses, and order of video segments. Therefore, when playing this type of streaming media in the browser:

We can directly use ffmpeg to slice the video. Run the command in the terminal.

```bash
ffmpeg -i input_video.mp4 \
-c:v libx264 \
-c:a aac \
-hls_time 10 \
-hls_segment_type mpegts \
-hls_list_size 0 \
-f hls \
-max_muxing_queue_size 1024 \
output.m3u8
```

In this command:

- `-i input_video.mp4` specifies the input video file.
- `-c:v libx264 -c:a aac` specifies the video and audio codecs.
- `-hls_time 10` specifies the duration of each M3U8 segment, in seconds. Here, the duration of each segment is set to 10 seconds.
- `-hls_segment_type mpegts` specifies the type of M3U8 segment as MPEG-TS.
- `-hls_list_size 0` sets the maximum number of segments included in the M3U8 file. Here, it is set to 0, indicating no limit.
- `-f hls` specifies the output format as HLS.
- `-max_muxing_queue_size 1024` sets the maximum multiplexing queue size to ensure that the output does not exceed the specified size.
- The final output file is `output.m3u8`.

Of course, there are more advanced uses of the ffmpeg slicing command, such as limiting the size of all segments to not exceed `500kb`, etc.

![As shown in the image, executing the above command, we generated some video segments and an m3u8 video list.](https://image-assets.dreams.plus/20240323234019.webp)

We open the m3u8 list file, which shows the order and duration of each segment as follows.

```plaintext
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:11.386378,
output0.ts
#EXTINF:11.011000,
output1.ts
#EXTINF:9.050711,
output2.ts
#EXTINF:8.591911,
output3.ts
#EXTINF:6.506933,
output4.ts
#EXT-X-ENDLIST
```

## Frontend Playback

After the browser gets the m3u8 file, it sequentially traverses this list and then loads the segments listed in the list in order, i.e., output0.ts → output1.ts ... → output4.ts. Since it loads and plays the video while downloading, it immediately starts playing the first segment after loading the first segment and then simultaneously loads the second segment, and so on.

Next, let's create a new HTML file in the same directory and use `video.js` to play the m3u8 list we just generated.

```html
<html>
  <head>
    <title>Play the video</title>
    <link href="https://vjs.zencdn.net/8.10.0/video-js.css" rel="stylesheet" />
  </head>
  <body>
    <video class="video-js" controls preload="auto" width="640" height="264">
      <source src="./output.m3u8" type="application/x-mpegURL" />
    </video>
    <script src="https://vjs.zencdn.net/8.10.0/video.min.js"></script>
  </body>
</html>
```

![](https://image-assets.dreams.plus/20240323235457.webp)

As you can see, the browser successfully loads and plays the segments listed in the list.

So far, the process of slicing and playing streaming media is roughly understood. Therefore, it is not difficult for us to implement the basic functionality of a streaming media server.

## Design API

Before we start, we need to design the upload and play APIs.

- Video upload: `/upload`, POST request, accepts a file in the `multipart/form-data` form.
- Video playback: `/play/{videoId}/video.m3u8`, GET request, returns a text data of type m3u8. Where `videoId` is the unique identifier for each video.

## Implement Upload Functionality

Now we will implement the video upload functionality for the server. Video uploading involves three stages: receiving the uploaded file → video slicing → video persistent storage. After uploading, return a video playback address to the client for video playback.

Front-end video uploading is generally done through form files. After the server receives the file, it slices the file and stores it in the public area, then returns the generated m3u8 file address.

First, initialize the node project and install the following packages: `express` is the server framework for Node.js, `multer` is a form file processing middleware developed for Express, `nanoid` generates a unique identifier for the uploaded resource, `fluent-ffmpeg` is one of the ffmpeg extensions set for Node.js, `serve` allows you to set up a

server access environment in the local directory.

```bash
npm i multer express nanoid fluent-ffmpeg cors
npm i serve --save-dev
```

Add the following fields to `package.json`:

```json
{
  "main": "main.js",
  "type": "module"
}
```

Then create the `main.js` file and write the following code to create a server:

```js
import express from "express";
import fs from "fs";
import multer from "multer";
import { nanoid } from "nanoid";
import ffmpeg from "fluent-ffmpeg";
import path from "path";
import cors from "cors";

const server = express();
server.listen(3300);
console.log("Server started.");
server.use(cors()); // For easy debugging, allow cross-origin
```

According to the designed API, add a POST route to upload videos:

```js
const copeUpload = multer({
  dest: "uploads-temp/",
  filename: function (req, file, cb) {
    const uniqueSuffix = nanoid();
    cb(null, file.fieldname + "-" + uniqueSuffix);
  },
}).single("video");

server.post("/upload", copeUpload, function (req, res, next) {
  const tempFilePath = path.resolve(req.file.path);
  const videoId = nanoid();
  const storageDirectory = path.resolve("storage", videoId);
  fs.mkdirSync(storageDirectory);

  ffmpeg(tempFilePath)
    .videoCodec("libx264")
    .audioCodec("aac")
    .addOption("-hls_time", 10)
    .addOption("-hls_segment_type", "mpegts")
    .addOption("-hls_list_size", 0)
    .format("hls")
    .addOption("-max_muxing_queue_size", 1024)
    .output(`${storageDirectory}/video.m3u8`)
    .on("start", function () {
      console.log("Starting video slice process.");
    })
    .on("end", function () {
      fs.rmSync(tempFilePath); // Delete the uploaded temporary file
      console.log("OK");
    })
    .on("error", function (err) {
      fs.rmSync(tempFilePath); // Delete the uploaded temporary file
      console.error("Error:", err);
    })
    .run();

  res.json(`http://localhost:3300/play/${videoId}/video.m3u8`); // Return the playback address
});
```

## Implement Playback Functionality

Next is to set up the playback API for the server.

After uploading the video, the server will return a playback address to the client, which is the download address of the m3u8 file. So this part is very simple.

When the player requests the video address `play/abcde/video.m3u8` with the ID `abcde`, since the m3u8 specifies that the slice file and the m3u8 file are in the same directory, the address for requesting the slice is also

```plaintext
play/abcde/output01.ts
play/abcde/output02.ts
play/abcde/output03.ts
......
```

So what the server actually needs to do is to retrieve the files requested by `/play/${videoId}/*` from the `storage` directory and return them to the client.

```js
server.get("/play/:videoId/:filename", (req, res) => {
  const videoId = req.params["videoId"];
  const storageDirectory = path.resolve("storage", videoId);
  if (!existsSync(storageDirectory)) {
    res.status(404).send();
  }
  const filename = req.params["filename"];
  const filepath = path.join(storageDirectory, filename);
  if (!existsSync(filepath)) {
    res.status(404).send();
  }
  const data = fs.readFileSync(filepath);
  res.send(data);
});
```

## Functional Testing

In the above process, our video server is completed in less than 100 lines of code. Next, we will write code for the client to test the upload and playback functions of the server.

Requirements: After uploading the video, return the video playback address and play the specified video address.

Create an `index.html` in the same directory:

```html
<html>
  <head>
    <title>Video Server</title>
    <link href="https://vjs.zencdn.net/8.10.0/video-js.css" rel="stylesheet" />
  </head>
  <body>
    <div>
      <h2>Upload Video Files</h2>
      <form id="uploadForm" method="POST" enctype="multipart/form-data">
        <input type="file" name="video" accept="video/*" />
        <button type="submit">Upload</button>
      </form>
      <div id="response"></div>
      <script>
        const form = document.getElementById("uploadForm");
        form.addEventListener("submit", function (event) {
          event.preventDefault();
          const formData = new FormData(form);
          fetch("http://localhost:3300/upload", {
            method: "POST",
            body: formData,
          })
            .then((response) => response.text())
            .then((data) => {
              document.getElementById("response").innerText = "Upload successful, video address is: " + data;
            })
            .catch((error) => {
              console.error("Request error:", error);
            });
        });
      </script>
    </div>

    <hr />

    <div>
      <h2>Play Video</h2>
      <label for="m3u8-url">Enter the m3u8 address of the video:</label>
      <input type="text" id="m3u8-url" name="m3u8-url" placeholder="For example: https://example.com/video.m3u8" />
      <button onclick="playVideo()">Play</button>

      <video id="my-video" class="video-js" controls preload="auto" width="640" height="360" data-setup="{}">
        <source src="" type="application/x-mpegURL" />
      </video>
      <script src="https://vjs.zencdn.net/8.10.0/video.min.js"></script>
      <script>
        function playVideo() {
          var m3u8Url = document.getElementById("m3u8-url").value;
          var videoPlayer = videojs("my-video");
          videoPlayer.src({
            src: m3u8Url,
            type: "application/x-mpegURL",
          });
          videoPlayer.play();
        }
      </script>
    </div>
  </body>
</html>
```

Then start the server:

```bash
node main.js
```

The server is running at `http://localhost:3300`.

Then start the user interface:

```bash
npx serve
```

You can access it in the browser at `http://localhost:3000`.

Test results are as follows:

![Test result](https://image-assets.dreams.plus/vdsafvdag.gif)

## Conclusion

In this article, we implemented a simple streaming media server and successfully implemented two core functions: video upload and video playback.

Of course, this project is just a demo. It is far from enough for

a video server used in a production environment to have only these two functions. It also requires other more powerful functions such as video resource management, resource health monitoring, network fluctuation monitoring, encryption, and security, etc. I won't go into details about these. If you are interested, you can implement them yourself.

## Download Source Code

The source code repository for this project is [here on GitHub](https://github.com/PrinOrange/simple-node-media-server). It's released under the CC 1.0 open-source license.


Node.js's non-blocking I/O and ffmpeg's video processing allow for building an efficient video streaming server. Let's implement a video server.

Let's Implement a Video Server with node.js and Ffmpeg


In software engineering, while module dependencies ideally form a **tree structure**, real-world implementations often adopt **Directed Acyclic Graphs (DAGs)** to enable practical code reuse. This article explores three fundamental patterns of module organization and their implications.

## Ideal Scenario: Hierarchical Tree Structure

### Characteristics of Tree Architecture

```plantuml
@startuml
top to bottom direction

rectangle "Core" as A
rectangle B
rectangle C
rectangle D
rectangle E
rectangle F

A --> B
A --> C
B --> D
B --> E
C --> F
@enduml
```

**Key Features:**  

- Unidirectional dependencies (parent → child)  
- Clear hierarchical separation (e.g., UI → Business Logic → Data Layer)  
- No circular references  

**Common Implementations:**  

- MVC Architecture  
- Clean Architecture  
- Three-Layer Pattern (Presentation/Logic/Data)

**Advantages:**  
✅ Straightforward dependency management  
✅ Explicit separation of concerns  
✅ Simplified testing and maintenance  

**Challenges:**  
⚠️ Limited code reuse opportunities  
⚠️ Potential for artificial module fragmentation  
⚠️ Suboptimal for complex systems with cross-cutting concerns  

## Practical Reality: Directed Acyclic Graph (DAG)

### Modern Dependency Management

```plantuml
@startuml
top to bottom direction

rectangle A
rectangle B
rectangle C
rectangle D
rectangle E
rectangle F

A --> B
A --> C
B --> D
B --> E
C --> F
C --> E
@enduml
```

**Key Features:**  

- Shared modules between components (E shared by B/C)  
- Multiple dependency paths  
- Strictly acyclic  

**Real-World Applications:**  

- React/Vue Component Systems  
- Webpack Module Federation  
- Bazel Build System  
- Database Schema Design  

**Advantages:**  
✅ Natural code reuse through shared modules  
✅ Flexible architecture for complex systems  
✅ Maintains dependency integrity  

**Implementation Considerations:**  
🔧 Requires robust tooling (TypeScript path resolution, Webpack aliases)  
🔧 Needs dependency visualization tools  
🔧 Demands strict code review practices  

## Anti-Pattern: Circular Dependencies

### Dangerous Dependency Loops

```plantuml
@startuml
top to bottom direction

rectangle A
rectangle B
rectangle C
rectangle D
rectangle E
rectangle F

A --> B
A --> C
B --> D
B --> E
C --> F
C --> E
E --> A
@enduml
```

**Consequences:**  
❌ Runtime initialization errors (`Cannot access 'X' before initialization`)  
❌ Compilation failures (ESM/TypeScript circular reference errors)  
❌ Unmaintainable "spaghetti code" structure  

**Common Culprits:**  

- Bidirectional controller-service relationships  
- Shared state management traps  
- Improper dependency injection  

## Architectural Strategies

### 1. Dependency Hoisting

Elevate shared dependencies to higher-level modules to prevent duplication while maintaining acyclicity.

### 2. Dependency Injection

Implement IoC containers to manage cross-module dependencies:

```typescript
// Angular-style DI example
@Injectable({ providedIn: 'root' })
export class SharedService {
  // Reusable business logic
}
```

### 3. Module Federation

Modern solutions for microfrontend architectures:

```javascript
// Webpack Module Federation Configuration
new ModuleFederationPlugin({
  name: 'app1',
  exposes: {
    './Widget': './src/components/Widget',
  }
})
```

## Industry Case Studies

| System Type        | Dependency Pattern | Notable Implementations       |
|---------------------|--------------------|-------------------------------|
| Frontend Frameworks | DAG                | React Component Tree, Vue SFC |
| Build Systems       | DAG                | Webpack, Turborepo, Bazel     |
| Microservices       | Tree/DAG Hybrid    | AWS Lambda Layers, NestJS     |
| Database Schemas    | DAG                | Foreign Key Relationships     |

## Best Practices Checklist

1. **Static Analysis**  
   - Enable TypeScript's `--noImplicitCircularity`  
   - Use ESLint `import/no-cycle` rule  

2. **Dependency Visualization**  

   ```bash
   npx madge --image graph.svg src/index.ts
   ```

3. **Architectural Guardrails**  
   - Enforce layer access rules in `tsconfig.json`  
   - Implement module boundary checks in CI pipelines

4. **Refactoring Techniques**  
   - Apply Dependency Inversion Principle (DIP)  
   - Utilize facade patterns for cross-module communication  

> "Good architecture maximizes the number of decisions not made."  
> ― Robert C. Martin (*Clean Architecture*)

By understanding these fundamental patterns and implementing appropriate guardrails, teams can create maintainable systems that balance architectural purity with practical reuse requirements.


In software engineering, the reference relationship of software modules should ideally be a tree, but in practical applications, in order to reuse certain module structures, a directed acyclic graph is often formed.

NOTE & BLOG

PROJECT-PRACTICE

Some Tips for Designing Tokenizer for Search Engine

Let's Implement a Video Server with node.js and Ffmpeg

How to Organize Modules in Software Architecture