Do Not Stand at My Grave and Weep

我没有离去


<div class="mt-10">
  <div class="flex justify-center my-5">
    <div>
      <span>Do not stand by my grave, and weep.</span><br/>
      <span>I am not there, I do not sleep—</span><br/><br/>
      <span>I am the thousand winds that blow </span><br/>
      <span>I am the diamond glints in snow </span><br/>
      <span>I am the sunlight on ripened grain, </span><br/>
      <span>I am the gentle, autumn rain. </span><br/>
      <span>As you awake with morning's hush, </span><br/>
      <span>I am the swift, up-flinging rush </span><br/>
      <span>Of quiet birds in circling flight, </span><br/>
      <span>I am the day transcending night. </span><br/><br/>
      <span>Do not stand by my grave, and cry— </span><br/>
      <span>I am not there, I did not die.</span><br/>
    </div>
  </div>
  <div class="text-right">
  — Clare Harner, The Gypsy,<br/>
  December 1934
  </div>
</div>

---

<div class="mb-10">
  <div class="flex justify-center my-5">
    <div>
      <span>不要在我的墓前哭泣，</span><br/>
      <span>我不在那里，我没有离去。</span><br/><br/>
      <span>我是扬起的万里清风，</span><br/>
      <span>我是白雪晶莹的光芒，</span><br/>
      <span>我是阳光下成熟的麦穗，</span><br/>
      <span>我是温柔的秋雨纷纷扬扬，</span><br/>
      <span>我是夜幕中闪烁的星光，</span><br/>
      <span>当你在晨曦中醒来时，</span><br/>
      <span>我是盘旋的鸟飞舞直上。</span><br/><br/>
      <span>不要在我的墓前哭泣，</span><br/>
      <span>我不在那里，我没有离去。</span><br/>
    </div>
  </div>
  <div class="text-right">
  — 程虹教授 译
  </div>
</div>



Node.js adopts an event-driven and non-blocking I/O model, allowing it to efficiently handle a large number of I/O requests and concurrent connections, which is crucial for a streaming media server. Meanwhile, FFmpeg, as a powerful video processing tool, provides rich video processing capabilities.

Combining Node.js's powerful extension capabilities, we can easily integrate FFmpeg to create a feature-complete video streaming media server, meeting various streaming media application requirements.

Now let's start from scratch and implement a simple video server: implementing the functionality to upload and play videos. In the process of learning, we will gradually understand how the video server works.

Before we begin, you need to:

- Understand the basic API of Node.js, such as the fs module, process module, etc.
- Understand the usage and common APIs of Express.js.
- Install and understand the basic usage of FFMPEG.

Now we want to implement a streaming media server. There are many streaming media protocols and formats on the market (RTP/RTCP/RTSP/RTMP/MMS/HLS, etc.), each with different applications. However, the vast majority of websites currently use HLS/M3U8 format streaming media, which is also supported by modern browsers.

Therefore, this article does not elaborate on these streaming media protocols but implements a simple HLS/M3U8 format streaming media server.

**The streaming media server implemented in this article is not technically difficult.**

## MP4 Video Slicing

Open some mainstream video websites, switch to the "Network" tab in Chrome Devtools, and then play the video. You can see that the browser is constantly downloading small files from a URL. These small files are video segments.

In general, web videos do not directly download the entire video file because a video file is usually several gigabytes, and downloading the entire file would be too large. Instead, the videos are split into many small segments, which are then continuously transferred to the browser. The browser then sequentially plays these segments.

For HLS/M3U8 type streaming media, the video playback address is the address for downloading the m3u8 file. The m3u8 file is a text-based list that stores the names, addresses, and order of video segments. Therefore, when playing this type of streaming media in the browser:

We can directly use ffmpeg to slice the video. Run the command in the terminal.

```bash
ffmpeg -i input_video.mp4 \
-c:v libx264 \
-c:a aac \
-hls_time 10 \
-hls_segment_type mpegts \
-hls_list_size 0 \
-f hls \
-max_muxing_queue_size 1024 \
output.m3u8
```

In this command:

- `-i input_video.mp4` specifies the input video file.
- `-c:v libx264 -c:a aac` specifies the video and audio codecs.
- `-hls_time 10` specifies the duration of each M3U8 segment, in seconds. Here, the duration of each segment is set to 10 seconds.
- `-hls_segment_type mpegts` specifies the type of M3U8 segment as MPEG-TS.
- `-hls_list_size 0` sets the maximum number of segments included in the M3U8 file. Here, it is set to 0, indicating no limit.
- `-f hls` specifies the output format as HLS.
- `-max_muxing_queue_size 1024` sets the maximum multiplexing queue size to ensure that the output does not exceed the specified size.
- The final output file is `output.m3u8`.

Of course, there are more advanced uses of the ffmpeg slicing command, such as limiting the size of all segments to not exceed `500kb`, etc.

![As shown in the image, executing the above command, we generated some video segments and an m3u8 video list.](https://raw.githubusercontent.com/PrinOrange/pic-host/master/myblog/20240323234019.webp)

We open the m3u8 list file, which shows the order and duration of each segment as follows.

```plaintext
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:11.386378,
output0.ts
#EXTINF:11.011000,
output1.ts
#EXTINF:9.050711,
output2.ts
#EXTINF:8.591911,
output3.ts
#EXTINF:6.506933,
output4.ts
#EXT-X-ENDLIST
```

## Frontend Playback

After the browser gets the m3u8 file, it sequentially traverses this list and then loads the segments listed in the list in order, i.e., output0.ts → output1.ts ... → output4.ts. Since it loads and plays the video while downloading, it immediately starts playing the first segment after loading the first segment and then simultaneously loads the second segment, and so on.

Next, let's create a new HTML file in the same directory and use `video.js` to play the m3u8 list we just generated.

```html
<html>
  <head>
    <title>Play the video</title>
    <link href="https://vjs.zencdn.net/8.10.0/video-js.css" rel="stylesheet" />
  </head>
  <body>
    <video class="video-js" controls preload="auto" width="640" height="264">
      <source src="./output.m3u8" type="application/x-mpegURL" />
    </video>
    <script src="https://vjs.zencdn.net/8.10.0/video.min.js"></script>
  </body>
</html>
```

![](https://raw.githubusercontent.com/PrinOrange/pic-host/master/myblog/20240323235457.webp)

As you can see, the browser successfully loads and plays the segments listed in the list.

So far, the process of slicing and playing streaming media is roughly understood. Therefore, it is not difficult for us to implement the basic functionality of a streaming media server.

## Design API

Before we start, we need to design the upload and play APIs.

- Video upload: `/upload`, POST request, accepts a file in the `multipart/form-data` form.
- Video playback: `/play/{videoId}/video.m3u8`, GET request, returns a text data of type m3u8. Where `videoId` is the unique identifier for each video.

## Implement Upload Functionality

Now we will implement the video upload functionality for the server. Video uploading involves three stages: receiving the uploaded file → video slicing → video persistent storage. After uploading, return a video playback address to the client for video playback.

Front-end video uploading is generally done through form files. After the server receives the file, it slices the file and stores it in the public area, then returns the generated m3u8 file address.

First, initialize the node project and install the following packages: `express` is the server framework for Node.js, `multer` is a form file processing middleware developed for Express, `nanoid` generates a unique identifier for the uploaded resource, `fluent-ffmpeg` is one of the ffmpeg extensions set for Node.js, `serve` allows you to set up a

server access environment in the local directory.

```bash
npm i multer express nanoid fluent-ffmpeg cors
npm i serve --save-dev
```

Add the following fields to `package.json`:

```json
{
  "main": "main.js",
  "type": "module"
}
```

Then create the `main.js` file and write the following code to create a server:

```js
import express from "express";
import fs from "fs";
import multer from "multer";
import { nanoid } from "nanoid";
import ffmpeg from "fluent-ffmpeg";
import path from "path";
import cors from "cors";

const server = express();
server.listen(3300);
console.log("Server started.");
server.use(cors()); // For easy debugging, allow cross-origin
```

According to the designed API, add a POST route to upload videos:

```js
const copeUpload = multer({
  dest: "uploads-temp/",
  filename: function (req, file, cb) {
    const uniqueSuffix = nanoid();
    cb(null, file.fieldname + "-" + uniqueSuffix);
  },
}).single("video");

server.post("/upload", copeUpload, function (req, res, next) {
  const tempFilePath = path.resolve(req.file.path);
  const videoId = nanoid();
  const storageDirectory = path.resolve("storage", videoId);
  fs.mkdirSync(storageDirectory);

  ffmpeg(tempFilePath)
    .videoCodec("libx264")
    .audioCodec("aac")
    .addOption("-hls_time", 10)
    .addOption("-hls_segment_type", "mpegts")
    .addOption("-hls_list_size", 0)
    .format("hls")
    .addOption("-max_muxing_queue_size", 1024)
    .output(`${storageDirectory}/video.m3u8`)
    .on("start", function () {
      console.log("Starting video slice process.");
    })
    .on("end", function () {
      fs.rmSync(tempFilePath); // Delete the uploaded temporary file
      console.log("OK");
    })
    .on("error", function (err) {
      fs.rmSync(tempFilePath); // Delete the uploaded temporary file
      console.error("Error:", err);
    })
    .run();

  res.json(`http://localhost:3300/play/${videoId}/video.m3u8`); // Return the playback address
});
```

## Implement Playback Functionality

Next is to set up the playback API for the server.

After uploading the video, the server will return a playback address to the client, which is the download address of the m3u8 file. So this part is very simple.

When the player requests the video address `play/abcde/video.m3u8` with the ID `abcde`, since the m3u8 specifies that the slice file and the m3u8 file are in the same directory, the address for requesting the slice is also

```
play/abcde/output01.ts
play/abcde/output02.ts
play/abcde/output03.ts
......
```

So what the server actually needs to do is to retrieve the files requested by `/play/${videoId}/*` from the `storage` directory and return them to the client.

```js
server.get("/play/:videoId/:filename", (req, res) => {
  const videoId = req.params["videoId"];
  const storageDirectory = path.resolve("storage", videoId);
  if (!existsSync(storageDirectory)) {
    res.status(404).send();
  }
  const filename = req.params["filename"];
  const filepath = path.join(storageDirectory, filename);
  if (!existsSync(filepath)) {
    res.status(404).send();
  }
  const data = fs.readFileSync(filepath);
  res.send(data);
});
```

## Functional Testing

In the above process, our video server is completed in less than 100 lines of code. Next, we will write code for the client to test the upload and playback functions of the server.

Requirements: After uploading the video, return the video playback address and play the specified video address.

Create an `index.html` in the same directory:

```html
<html>
  <head>
    <title>Video Server</title>
    <link href="https://vjs.zencdn.net/8.10.0/video-js.css" rel="stylesheet" />
  </head>
  <body>
    <div>
      <h2>Upload Video Files</h2>
      <form id="uploadForm" method="POST" enctype="multipart/form-data">
        <input type="file" name="video" accept="video/*" />
        <button type="submit">Upload</button>
      </form>
      <div id="response"></div>
      <script>
        const form = document.getElementById("uploadForm");
        form.addEventListener("submit", function (event) {
          event.preventDefault();
          const formData = new FormData(form);
          fetch("http://localhost:3300/upload", {
            method: "POST",
            body: formData,
          })
            .then((response) => response.text())
            .then((data) => {
              document.getElementById("response").innerText = "Upload successful, video address is: " + data;
            })
            .catch((error) => {
              console.error("Request error:", error);
            });
        });
      </script>
    </div>

    <hr />

    <div>
      <h2>Play Video</h2>
      <label for="m3u8-url">Enter the m3u8 address of the video:</label>
      <input type="text" id="m3u8-url" name="m3u8-url" placeholder="For example: https://example.com/video.m3u8" />
      <button onclick="playVideo()">Play</button>

      <video id="my-video" class="video-js" controls preload="auto" width="640" height="360" data-setup="{}">
        <source src="" type="application/x-mpegURL" />
      </video>
      <script src="https://vjs.zencdn.net/8.10.0/video.min.js"></script>
      <script>
        function playVideo() {
          var m3u8Url = document.getElementById("m3u8-url").value;
          var videoPlayer = videojs("my-video");
          videoPlayer.src({
            src: m3u8Url,
            type: "application/x-mpegURL",
          });
          videoPlayer.play();
        }
      </script>
    </div>
  </body>
</html>
```

Then start the server:

```bash
node main.js
```

The server is running at `http://localhost:3300`.

Then start the user interface:

```bash
npx serve
```

You can access it in the browser at `http://localhost:3000`.

Test results are as follows:

![Test result](https://raw.githubusercontent.com/PrinOrange/pic-host/master/myblog/vdsafvdag.gif)

## Conclusion

In this article, we implemented a simple streaming media server and successfully implemented two core functions: video upload and video playback.

Of course, this project is just a demo. It is far from enough for

a video server used in a production environment to have only these two functions. It also requires other more powerful functions such as video resource management, resource health monitoring, network fluctuation monitoring, encryption, and security, etc. I won't go into details about these. If you are interested, you can implement them yourself.

## Download Source Code

The source code repository for this project is [here on GitHub](https://github.com/PrinOrange/simple-node-media-server). It's released under the CC 1.0 open-source license.


Node.js utilizes an event-driven and non-blocking I/O model, which enables it to efficiently handle a large number of I/O requests and concurrent connections. This is crucial for a streaming media server. Meanwhile, ffmpeg provides powerful functionality for video processing. We can leverage Node.js's powerful extension capabilities to integrate ffmpeg and create a video streaming media server.

Let's Implement a Video Server with node.js and Ffmpeg


For the integration of piecewise continuous functions, the results of "rectangular approximation" and "trapezoidal approximation" are the same, with the difference lying only in the speed of approximation. The essence of this problem lies in "error estimation".

We only need to analyze the "error reduction process" associated with the "approximation process".

To better understand the error, I will specifically calculate and analyze the integrations using rectangular and trapezoidal approximations for continuous functions on closed intervals.

Given $f \in \mathcal{C}[a, b]$, let's start with the expression for rectangular approximation integration:

$$
\lim_{n\to\infty}\sum_{k=1}^n f\left(\frac{(n-k)a+kb}{n}\right)\cdot\frac{(a-b)}{n} = \\
\lim_{n\to\infty}\sum_{k=1}^{n-1} f\left(\frac{(n-k)a+kb}{n}\right)\cdot\frac{(a-b)}{n}+f(b)\cdot\frac{(a-b)}{n} \quad {(1)}
$$

Now, let's look at the expression for trapezoidal approximation:

$$
\lim_{n\to\infty}\sum_{k=1}^n\frac{f\left(\frac{(n-k+1)a+(k-1)b}{n}\right)+f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n}\dots \quad {(2)}
$$

Let's rewrite the above expression, **pay attention to the change in the upper limit of summation**, observe:

$$
\begin{align*}
&\lim_{n\to\infty}\sum_{k=1}^n\frac{f\left(\frac{(n-k+1)a+(k-1)b}{n}\right)+f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n} \\
&=\lim_{n\to\infty}\sum_{k=1}^n\frac{f\left(\frac{(n-k+1)a+(k-1)b}{n}\right)}{2}\cdot\frac{(a-b)}{n} \\
&\qquad\quad +\lim_{n\to\infty}\sum_{k=1}^n\frac{f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n} \\
&=\lim_{n\to\infty}\sum_{k=0}^{n-1}\frac{f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n} \\
&\qquad\quad +\lim_{n\to\infty}\sum_{k=1}^n\frac{f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n} \\
&=\lim_{n\to\infty}\left[\left(\frac{f(a)}{2}\cdot\frac{(a-b)}{n}+\sum_{k=1}^{n-1}\frac{f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n}\right) \right. \\
&\qquad\quad \left. +\left(\sum_{k=1}^{n-1}\frac{f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n}+\frac{f(b)}{2}\cdot\frac{(a-b)}{n}\right)\right] \\
&=\lim_{n\to\infty}(\frac{f(a)}{2}\cdot\frac{(a-b)}{n}+\frac{f(b)}{2}\cdot\frac{(a-b)}{n})+ \\
& \lim_{n\to\infty}(2\cdot\sum_{k=1}^{n-1}\frac{f\left(\frac{(n-k)a+kb}{n}\right)}{2}\cdot\frac{(a-b)}{n}) \\
&=\lim_{n\to\infty}(\frac{f(a)+f(b)}{2}\cdot\frac{(a-b)}{n}+\sum_{k=1}^{n-1}f(\frac{(n-k)a+kb}{n})\cdot\frac{(a-b)}{n}) \\
&=\lim_{n\to\infty}\sum_{k=1}^{n-1}f\left(\frac{(n-k)a+kb}{n}\right)\cdot\frac{(a-b)}{n}+\frac{f(a)+f(b)}{2}\cdot\frac{(a-b)}{n} \dots \quad {(3)}
\end{align*}
$$

Comparing equation ${3}$ with equation ${1}$, did you notice? When integrating using two different approximation methods, the difference lies in:

$$
\frac{f(a)+f(b)}{2}\cdot\frac{(a-b)}{n}-f(b)\cdot\frac{(a-b)}{n} \\
=\frac{f(a)-f(b)}{2}\cdot\frac{(a-b)}{n}
$$

We denote this expression as $\Delta(n)$. Notice, it only involves one variable $n$. And it's easy to see:

- $I.$ If $f(a)=f(b)$, then $\Delta(n)\equiv0$.

- $II.$ If $f(a)\neq f(b)$, then $\Delta(n)=O(\frac{1}{n})$.

The first case might give you a feeling of reading an O. Henry novel. Surprisingly, only if $f(a)=f(b)$, no matter how the graph of $f$ fluctuates within the interval, rectangular and trapezoidal approximations are completely consistent, resulting in equal errors. As expected, when you write out the two expressions and observe each term of the summation, you'll find that the terms in both expressions are roughly the same.

The second case leads to a deeper conclusion. Let's denote the error of equation $(1)$ as $\delta_1(n)$ and the error of equation $(3)$ as $\delta_2(n)$, we have:

$$
\delta_1(n)-\delta_2(n)=\Delta(n)=O(\frac{1}{n})
$$

What does this mean? It means that $\delta_1(n)$ and $\delta_2(n)$ are at most one of them has an error term smaller than $o(\frac{1}{n})$! From your intuition, you might have guessed that the error of the trapezoidal approximation is smaller. Correct, the conclusion was proved.


Discusses the equivalence and error estimation in integrating piecewise continuous functions using rectangular and trapezoidal approximations.

NOTE & BLOG