A tensor is a higher-dimensional generalization of a scalar, a vector, and a matrix

You have almost certainly encountered this sentence. It appears on the first page of virtually every textbook, and it is also the point of departure for virtually all confusion.

The reason is that this sentence describes only the storage format of a tensor, while saying nothing whatsoever about its essence. It is akin to stating that "a human being is a collection of cells"—not strictly false, yet it does not answer the genuine question of what a human being is.

This article pursues a single objective: why are physicists and mathematicians so insistent on distinguishing vectors from "dual vectors," and on distinguishing "upper indices" from "lower indices" in tensors? If you can grasp this one point, all the mystique surrounding tensors evaporates.

We shall proceed in the following order: first, we reexamine the vector itself; next, we introduce its twin sibling, the dual vector; then we furnish a rigorous definition of a tensor; we dismantle a common point of confusion with a concrete example; and finally, we elucidate the critical bridge connecting vectors and dual vectors—the metric.

Arrows

Suppose there is a physical arrow lying on a plane, pointing 30° east of north, with a length of 5 meters. This arrow has an objective existence—no matter how you erect a coordinate system, it remains exactly where it is; its length and direction do not alter merely because you have changed your frame of reference.

Yet when you write down coordinates such as (3, 4), you have performed a concealed operation: you have selected a coordinate basis and subsequently "translated" the arrow into a tuple of numbers with respect to that basis. Switch to a different basis (for instance, by rotating the coordinate axes by 45°), and the very same arrow is translated into an entirely different set of numbers, say (4.95, 0.71).

The numbers changed; the arrow did not. This is precisely the most fundamental distinction between a vector as a geometric object and an array as its coordinate representation. A tensor in deep learning (e.g., torch.tensor([3,4])) is merely this "translation result"; it never concerns itself with whether an invariant geometric object stands behind it. This is perfectly acceptable when one is engaged in purely numerical computation, but the moment one wishes to entertain propositions such as "physical laws do not depend on the choice of observer," one must return to the arrow itself.

Setting coordinates aside, the definition of a vector space $V$ is in fact remarkably parsimonious: so long as a collection of objects supports addition (tip-to-tail composition) and scalar multiplication (stretching or shrinking), and satisfies a handful of natural algebraic laws, it constitutes a vector space, and its elements are vectors. Velocity, force, and displacement are all objects of this kind.

Dual Space

A vector is an arrow; what, then, is the instrument that "measures" that arrow?

It is an object of an altogether different type, called a dual vector (also termed a covector or a 1-form). It is not an arrow but rather a ruler—a family of equally spaced parallel lines. When you lay the arrow upon it, the number of lines the arrow crosses is the number it outputs.

A rigorous definition: the dual space $V^*$ is the set of all linear functionals on $V$. That is, each element $\omega$ in $V^*$ is a rule: feed it a vector $\mathbf{v}$, and it emits a real number $\omega(\mathbf{v})$, and this rule is linear with respect to vector addition and scalar multiplication.

In finite-dimensional linear algebra, this is so straightforward as to be almost banal:

• A vector $\mathbf{v}$ is a column vector: $\begin{bmatrix} 2 \\ 3 \end{bmatrix}$
• A dual vector $\omega$ is a row vector: $\begin{bmatrix} 1 & 4 \end{bmatrix}$

Their "pairing" is simply matrix multiplication:

The space formed by all row vectors is precisely the dual space of the column-vector space. This sounds as though the same set of numbers has merely been rearranged—but when the coordinate system undergoes a transformation, the "fates" of these two entities diverge dramatically, and herein lies the crux of the matter.

Imagine changing the unit of measurement from meters to centimeters (which amounts to shrinking the coordinate basis by a factor of $1/100$).

• Measured in centimeters, the vector components are magnified by a factor of 100 (the same arrow, with a smaller unit, requires a larger count of tick marks).
• The dual-vector components, however (that is, the ruler-density quantity "how many units correspond to one tick"), are shrunk by a factor of 1/100 (the tick lines themselves become denser; the density as such is unchanged, but the number representing "how many ticks per unit" changes).

Multiply the two together: $100 \times \frac{1}{100} = 1$, and the pairing result—that real number—remains invariant.

This is the origin of the physical terminology of "contravariant" (vectors, whose components transform opposite to the basis) and "covariant" (dual vectors, whose components transform in the same direction as the basis). The entire raison d'être of the dual space is to guarantee that the number ultimately computed—be it a length, a work, an energy, or a probability—is an objective fact, independent of whether one chooses meters or centimeters, or which coordinate system one adopts.

This is also the single most important sentence in the entire article: vectors and dual vectors form a complementary pair; their respective components vary with the choice of coordinates, yet the scalar obtained by pairing them is forever invariant. Every technical detail of tensor theory is, at bottom, in the service of preserving this invariance.

The Rigorous Definition of a Tensor

Once vectors and dual vectors are in hand, the definition of a tensor emerges naturally:

A tensor $T$ of type $(p, q)$ is a multilinear map: it simultaneously accepts $p$ dual vectors and $q$ vectors as arguments and outputs a real number.

In symbols:

Two keywords, neither dispensable:

• Multilinear: linear in each slot individually (additive and homogeneous with respect to scalar multiplication). This guarantees that tensor operations can be fully expanded as weighted sums of components, just as matrix operations can.
• Scalar output: irrespective of the number of slots or the variety of input types, what ultimately emerges is an absolute number that does not vary with changes of coordinates.

From this we directly obtain the formula for the order of a tensor:

This is also why the term "order" alone carries insufficient information—it tells you how many slots a tensor possesses, but not which slots consume vectors and which consume dual vectors. This distinction is by no means a verbal quibble; the following chapter will illustrate, with a concrete example, why it is a matter of life and death.

The Slot Philosophy

Physics textbooks frequently describe second-order tensors in the following manner:

"The stress tensor $\sigma$ acts on a normal vector $\mathbf{n}$ to produce a stress vector $\mathbf{t}$."

This sentence sounds like "input a vector, output a vector." Yet the definition in Chapter 3 explicitly states that a tensor is something that "inputs several vectors and dual vectors and outputs a scalar." The two statements appear utterly irreconcilable—and this is precisely where the vast majority of readers become stuck.

The key that resolves this contradiction is: a multilinear map does not require you to fill all of its slots at once.

The stress tensor $\sigma$ is a tensor of type $(1,1)$, possessing two slots: one dual-vector slot and one vector slot. When you insert only the normal vector $\mathbf{n}$ into its vector slot:

• The vector slot is now filled;
• The dual-vector slot remains vacant.

What has $\sigma(\cdot, \mathbf{n})$ become? It has become an intermediate product still awaiting a dual-vector input—and "a machine that awaits a dual vector and then emits a scalar" happens to be another rigorous way of defining a vector (via the natural isomorphism $V \cong V^{**}$).

Thus:

• Only one slot filled: $\sigma$ degenerates into a "pending" vector, which is precisely the "output stress vector $\mathbf{t}$" referred to in textbooks;
• Both slots filled (the normal vector + a dual vector / ruler along some direction): the tensor finally emits a scalar, namely the magnitude of the stress component in that direction.

Hence "input a vector, output a vector" and "input two arguments, output a scalar" are not at all contradictory; the former is merely an intermediate state of the latter. Physics textbooks, privileging intuition, typically exhibit only the "half-fed" result; the mathematical definition rigorously describes the ultimate behavior once all slots are filled. Once this layer is understood, virtually all disputes over whether a tensor "is a mapping or an operation" dissolve of their own accord.

Why, then, must $p$ and $q$ be distinguished, rather than merely speaking of the order?

A tensor of type $(1,1)$ and a tensor of type $(0,2)$, when written out as arrays, are both two-dimensional matrices—visually indistinguishable. Yet their behavior under coordinate transformations is altogether different:

• Type $(1,1)$ (e.g., linear transformations, the inertia tensor): one upper index and one lower index; under transformation one component is "contravariant" and the other "covariant," and the two cancel each other out.
• Type $(0,2)$ (e.g., the metric tensor): two lower indices; under transformation both components are "covariant" and accumulate in the same direction.

If one speaks only of "order" while neglecting the $(p, q)$ distinction, one will inevitably misapply formulas when computing operations such as divergence or contraction—and in general relativity, an error of this kind is sufficient to cause a physical law such as "conservation of energy" to collapse computationally. This is why mathematicians insist on labeling tensors by their $(p,q)$ type rather than by the catch-all term "order."

The Metric Tensor

One question remains unanswered at this stage: vectors and dual vectors reside in two manifestly distinct spaces, yet physicists routinely treat them interchangeably (for instance, speaking of "lowering an index on a vector"). What is going on?

In a space equipped with a metric tensor $g_{ij}$ (as nearly all physical spaces are), a natural "translator" exists, capable of translating a vector into a dual vector, and vice versa. This operation is called index raising and lowering:

Lowering an index: vector to dual vector

Raising an index: dual vector to vector

The metric tensor itself is a tensor of type $(0,2)$; its role is precisely to "ingest two vectors and emit their inner product"—and the inner product itself is what defines length and angle. It is only by virtue of the metric that we can speak of "the length of a vector," a quantity that sounds as though it were an intrinsic property of the vector but in fact cannot be defined without recourse to a dual pairing.

In Euclidean space (such as the familiar Cartesian coordinate system), the metric happens to be the identity matrix, and index raising and lowering "appear" to do nothing at all. This is also why, in secondary-school physics, the distinction between vectors and dual vectors is entirely concealed. The moment the coordinate system becomes more involved (polar coordinates, spherical coordinates), or the space itself is curved (the spacetime of general relativity), the metric ceases to be the identity matrix, and the distinction between vectors and dual vectors reasserts itself with decisive importance.

The Coordinate Transformation Rule

The preceding chapters have all appealed to intuition; here we provide the concrete component transformation rule as an "anchor." Let coordinates transform from $x$ to $x'$. The component transformation rule for a tensor of type $(p,q)$ is:

There is no need to be intimidated by this formula; it does nothing more than reiterate a single point:

• Every upper index (corresponding to a "contravariant" component of a vector slot) is multiplied by a Jacobian factor of the form $\frac{\partial x'}{\partial x}$;
• Every lower index (corresponding to a "covariant" component of a dual-vector slot) is multiplied by a Jacobian factor of the form $\frac{\partial x}{\partial x'}$ (which is precisely the inverse of the former).

This is the rigorous mathematical version of the intuitive example from Chapter 2—"centimeters to meters": the Jacobian factors for upper and lower indices are mutual inverses; when paired and contracted they cancel one another exactly, thereby ensuring that the tensor as a whole (qua geometric object) remains invariant, with only its components relative to a particular coordinate system undergoing change.

Common Tensors

Below are examples of tensors encountered across a variety of domains.

Two Registers and Their Rules

A final clarification is in order: the word "tensor" as used in deep learning and as used in physics / differential geometry refers to concepts at two distinct levels of abstraction, and this is a fertile source of confusion.

• In deep learning: a tensor is simply a multidimensional array; the sole concern is the efficiency of parallel computation. There is no such thing as "invariance under coordinate transformation," nor any distinction between covariance and contravariance. Here, understanding a tensor as a "generalized array" is entirely sufficient; there is no need to overburden the term with philosophical weight.
• In physics and differential geometry: a tensor is first and foremost a geometric or physical object that does not depend on any choice of coordinates; an array is merely its "provisional snapshot" with respect to a particular coordinate basis. Covariance, contravariance, the metric, contraction—these concepts exist for one purpose and one purpose only: to guarantee that the physical quantities computed from these provisional snapshots (length, energy, curvature) are objective, absolute, and invariant under changes of the observer.

Neither register is "right" or "wrong"; they serve radically different ends. Genuine confusion tends to arise from using deep-learning intuition to understand the tensors of general relativity, or vice versa. Once one recognizes that these are two distinct enterprises, much of the frustration that takes the form of "the textbook explains this poorly" dissolves of its own accord.

Conclusion

Let us return to the original question: what, after all, is a tensor?

It is a machine furnished with a certain number of "slots": some slots await "arrows" (vectors), others await "rulers" (dual vectors). When every slot has been filled, the number it emits is forever the same—regardless of the coordinate system from which you observe it, regardless of the units in which you measure. The metric tensor, for its part, is the bridge connecting the world of vectors to the world of dual vectors, making it possible to define rigorously a notion as seemingly innocent as "length."

The next time you write down a lower index or an upper index, you might pause to consider: is this position waiting for an arrow, or is it waiting for a ruler?