I sometimes see people refer to neural networks as just "another tool in your machine learning toolbox".
我时常看到有人把神经网络说成"你机器学习工具箱里的又一件工具"。
They have some pros and cons, they work here or there, and sometimes you can use them to win Kaggle competitions.
它们有优点也有缺点，这里那里能用一用，有时还能拿去赢个 Kaggle 比赛。
Unfortunately, this interpretation completely misses the forest for the trees.
遗憾的是，这种理解完全是只见树木、不见森林。
Neural networks are not just another classifier, they represent the beginning of a fundamental shift in how we develop software.
神经网络不只是又一种分类器，它们标志着我们开发软件的方式正在发生一场根本性转变的开端。
They are Software 2.0.
它们是软件 2.0。
The "classical stack" of Software 1.0 is what we're all familiar with — it is written in languages such as Python, C++, etc.
软件 1.0 的"经典技术栈"是我们人人熟悉的那套——用 Python、C++ 之类的语言写成。
It consists of explicit instructions to the computer written by a programmer.
它由程序员写下的、给计算机的显式指令组成。
By writing each line of code, the programmer identifies a specific point in program space with some desirable behavior.
程序员每写一行代码，就是在程序空间中钉下一个具有某种期望行为的具体的点。
In contrast, Software 2.0 is written in much more abstract, human unfriendly language, such as the weights of a neural network.
与之相反，软件 2.0 用一种抽象得多、对人类很不友好的语言写成，比如神经网络的权重（weights）。
No human is involved in writing this code because there are a lot of weights (typical networks might have millions), and coding directly in weights is kind of hard (I tried).
没有任何人参与编写这种代码，因为权重实在太多（典型的网络可能有几百万个），而直接用权重来编程相当困难（我试过）。
Instead, our approach is to specify some goal on the behavior of a desirable program (e.g., "satisfy a dataset of input output pairs of examples", or "win a game of Go"), write a rough skeleton of the code (i.e. a neural net architecture) that identifies a subset of program space to search, and use the computational resources at our disposal to search this space for a program that works.
我们的做法是：先给期望的程序行为定一个目标（比如"拟合一个输入-输出样例对的数据集"，或"赢一盘围棋"），再写一个粗略的代码骨架（即神经网络架构）来圈定要搜索的程序空间子集，然后动用手头的算力，在这个空间里搜出一个能用的程序。
In the case of neural networks, we restrict the search to a continuous subset of the program space where the search process can be made (somewhat surprisingly) efficient with backpropagation and stochastic gradient descent.
对神经网络而言，我们把搜索限制在程序空间的一个连续子集里，在这里搜索过程可以借助反向传播（backpropagation）和随机梯度下降（SGD）变得（有点出人意料地）高效。
To make the analogy explicit, in Software 1.0, human-engineered source code (e.g. some .cpp files) is compiled into a binary that does useful work.
把类比说透：在软件 1.0 里，人类手工打造的源代码（比如一些 .cpp 文件）被编译成一个干活的二进制程序。
In Software 2.0 most often the source code comprises 1) the dataset that defines the desirable behavior and 2) the neural net architecture that gives the rough skeleton of the code, but with many details (the weights) to be filled in.
在软件 2.0 里，"源代码"通常由两部分组成：1）定义期望行为的数据集；2）给出代码粗略骨架的神经网络架构——但大量细节（权重）留待填充。
The process of training the neural network compiles the dataset into the binary — the final neural network.
训练神经网络的过程，就是把数据集"编译"成二进制产物——最终的那个神经网络。
In most practical applications today, the neural net architectures and the training systems are increasingly standardized into a commodity, so most of the active "software development" takes the form of curating, growing, massaging and cleaning labeled datasets.
在今天的多数实际应用中，网络架构和训练系统正日益标准化为大路货，所以真正活跃的"软件开发"，形式上变成了对带标注数据集的筛选、扩充、揉捏与清洗。
This is fundamentally altering the programming paradigm by which we iterate on our software, as the teams split in two: the 2.0 programmers (data labelers) edit and grow the datasets, while a few 1.0 programmers maintain and iterate on the surrounding training code infrastructure, analytics, visualizations and labeling interfaces.
这从根本上改变了我们迭代软件的编程范式：团队一分为二——2.0 程序员（数据标注者）编辑和扩充数据集，少数 1.0 程序员维护和迭代周边的训练基础设施、分析、可视化与标注界面。
It turns out that a large portion of real-world problems have the property that it is significantly easier to collect the data (or more generally, identify a desirable behavior) than to explicitly write the program.
事实证明，现实世界的一大部分问题都有这样的性质：收集数据（或更宽泛地说，指认出期望的行为）远比显式地写出程序容易。
Because of this and many other benefits of Software 2.0 programs that I will go into below, we are witnessing a massive transition across the industry where of a lot of 1.0 code is being ported into 2.0 code.
因为这一点，加上下文将展开的软件 2.0 的诸多其他好处，我们正在见证一场席卷全行业的大迁移：大量 1.0 代码被移植成 2.0 代码。
Software (1.0) is eating the world, and now AI (Software 2.0) is eating software.
软件（1.0）正在吞噬世界，而现在，AI（软件 2.0）正在吞噬软件。
Let's briefly examine some concrete examples of this ongoing transition.
我们简要看几个这场迁移正在发生的具体例子。
In each of these areas we've seen improvements over the last few years when we give up on trying to address a complex problem by writing explicit code and instead transition the code into the 2.0 stack.
在下面每个领域，过去几年我们都看到了同一件事：一旦放弃用显式代码硬解复杂问题、把代码迁进 2.0 技术栈，效果就提升。
Visual Recognition used to consist of engineered features with a bit of machine learning sprinkled on top at the end (e.g., an SVM).
视觉识别（Visual Recognition）过去由人工设计的特征组成，最后再撒一点机器学习（比如一个 SVM）在上面。
Since then, we discovered much more powerful visual features by obtaining large datasets (e.g. ImageNet) and searching in the space of Convolutional Neural Network architectures.
此后，靠着获取大数据集（如 ImageNet）并在卷积神经网络（CNN）架构空间中搜索，我们发现了强大得多的视觉特征。
More recently, we don't even trust ourselves to hand-code the architectures and we've begun searching over those as well.
最近，我们甚至不再信任自己手写架构的能力，连架构本身也开始交给搜索。
Speech recognition used to involve a lot of preprocessing, gaussian mixture models and hidden markov models, but today consist almost entirely of neural net stuff.
语音识别（Speech recognition）过去要做大量预处理，用高斯混合模型和隐马尔可夫模型，如今几乎全盘换成了神经网络的东西。
A very related, often cited humorous quote attributed to Fred Jelinek from 1985 reads "Every time I fire a linguist, the performance of our speech recognition system goes up".
与此密切相关、常被引用的一句幽默名言，据传出自 Fred Jelinek（1985 年）："我每解雇一个语言学家，语音识别系统的性能就涨一截。"
Speech synthesis has historically been approached with various stitching mechanisms, but today the state of the art models are large ConvNets (e.g. WaveNet) that produce raw audio signal outputs.
语音合成（Speech synthesis）历来靠各种拼接机制来做，而今天最先进的模型是直接输出原始音频信号的大型卷积网络（如 WaveNet）。
Machine Translation has usually been approaches with phrase-based statistical techniques, but neural networks are quickly becoming dominant.
机器翻译（Machine Translation）过去通常用基于短语的统计技术，而神经网络正迅速占据主导。
My favorite architectures are trained in the multilingual setting, where a single model translates from any source language to any target language, and in weakly supervised (or entirely unsupervised) settings.
我最喜欢的架构是在多语言设定下训练的：单一模型从任意源语言翻到任意目标语言，且可在弱监督（甚至完全无监督）的设定下训练。
Games. Explicitly hand-coded Go playing programs have been developed for a long while, but AlphaGo Zero (a ConvNet that looks at the raw state of the board and plays a move) has now become by far the strongest player of the game.
游戏。手工编写的围棋程序开发了很久，但 AlphaGo Zero（一个直接看棋盘原始状态然后落子的卷积网络）如今已是这项游戏遥遥领先的最强棋手。
I expect we're going to see very similar results in other areas, e.g. DOTA 2, or StarCraft.
我预计在其他领域会看到非常相似的结果，比如 DOTA 2 或星际争霸。
Databases. More traditional systems outside of Artificial Intelligence are also seeing early hints of a transition.
数据库。AI 之外的更传统系统，也开始出现迁移的早期迹象。
For instance, "The Case for Learned Index Structures" replaces core components of a data management system with a neural network, outperforming cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory.
例如《The Case for Learned Index Structures（为可学习索引结构辩护）》一文用神经网络替换了数据管理系统的核心组件，速度比缓存优化过的 B 树最高快 70%，内存还省一个数量级。
You'll notice that many of my links above involve work done at Google.
你会注意到，上面很多链接都指向 Google 的工作。
This is because Google is currently at the forefront of re-writing large chunks of itself into Software 2.0 code.
这是因为 Google 眼下正站在把自身大块代码重写为软件 2.0 的最前沿。
"One model to rule them all" provides an early sketch of what this might look like, where the statistical strength of the individual domains is amalgamated into one consistent understanding of the world.
《One model to rule them all（一个模型统治一切）》给出了这幅图景的早期草稿：各个领域的统计力量被熔铸成对世界的一份统一理解。
Why should we prefer to port complex programs into Software 2.0?
为什么我们应该倾向于把复杂程序移植成软件 2.0？
Clearly, one easy answer is that they work better in practice.
显然，一个现成的答案是：它们在实践中效果更好。
However, there are a lot of other convenient reasons to prefer this stack.
但除此之外，偏爱这套技术栈还有许多顺手的理由。
Let's take a look at some of the benefits of Software 2.0 (think: a ConvNet) compared to Software 1.0 (think: a production-level C++ code base). Software 2.0 is:
来看看软件 2.0（想象一个卷积网络）相对软件 1.0（想象一个生产级 C++ 代码库）的一些好处。软件 2.0 具有：
Computationally homogeneous. A typical neural network is, to the first order, made up of a sandwich of only two operations: matrix multiplication and thresholding at zero (ReLU).
计算同质性。一阶近似地看，典型神经网络只由两种运算三明治式地叠成：矩阵乘法，和过零阈值（ReLU）。
Compare that with the instruction set of classical software, which is significantly more heterogenous and complex.
对比经典软件的指令集——那要异质和复杂得多。
Because you only have to provide Software 1.0 implementation for a small number of the core computational primitives (e.g. matrix multiply), it is much easier to make various correctness/performance guarantees.
因为你只需为少数核心计算原语（如矩阵乘法）提供软件 1.0 实现，各种正确性与性能保证也就容易做得多。
Simple to bake into silicon. As a corollary, since the instruction set of a neural network is relatively small, it is significantly easier to implement these networks much closer to silicon, e.g. with custom ASICs, neuromorphic chips, and so on.
易于烧进芯片。作为推论，既然神经网络的指令集相对很小，把这些网络实现得离硅更近就容易得多，比如定制 ASIC、神经形态芯片等等。
The world will change when low-powered intelligence becomes pervasive around us.
当低功耗智能在我们身边无处不在时，世界将为之改变。
E.g., small, inexpensive chips could come with a pretrained ConvNet, a speech recognizer, and a WaveNet speech synthesis network all integrated in a small protobrain that you can attach to stuff.
比如，小而便宜的芯片可以内置预训练的卷积网络、语音识别器和 WaveNet 语音合成网络，集成为一个可以贴在任何东西上的小小"原型脑"。
Constant running time. Every iteration of a typical neural net forward pass takes exactly the same amount of FLOPS.
恒定运行时间。典型神经网络每次前向传播消耗的浮点运算量（FLOPS）完全相同。
There is zero variability based on the different execution paths your code could take through some sprawling C++ code base.
不存在代码在庞杂 C++ 库里走不同执行路径带来的任何波动。
Of course, you could have dynamic compute graphs but the execution flow is normally still significantly constrained.
当然，你可以用动态计算图，但执行流通常仍受很强的约束。
This way we are also almost guaranteed to never find ourselves in unintended infinite loops.
这样一来，我们也几乎保证永远不会落进意料之外的死循环。
Constant memory use. Related to the above, there is no dynamically allocated memory anywhere so there is also little possibility of swapping to disk, or memory leaks that you have to hunt down in your code.
恒定内存占用。与上一条相关：任何地方都没有动态分配的内存，因此几乎不可能换页到磁盘，也几乎没有需要你在代码里追杀的内存泄漏。
It is highly portable. A sequence of matrix multiplies is significantly easier to run on arbitrary computational configurations compared to classical binaries or scripts.
高度可移植。一串矩阵乘法，比经典二进制程序或脚本容易得多地跑在任意计算配置上。
It is very agile. If you had a C++ code and someone wanted you to make it twice as fast (at cost of performance if needed), it would be highly non-trivial to tune the system for the new spec.
非常敏捷。如果你有一份 C++ 代码，有人要你把它加速一倍（必要时可牺牲效果），要为新指标调整系统绝非易事。
However, in Software 2.0 we can take our network, remove half of the channels, retrain, and there — it runs exactly at twice the speed and works a bit worse.
但在软件 2.0 里，我们可以拿起网络、砍掉一半通道、重新训练——好了，它正好快一倍，效果稍差一点。
It's magic.
这就是魔法。
Conversely, if you happen to get more data/compute, you can immediately make your program work better just by adding more channels and retraining.
反过来，如果你恰好拿到更多数据或算力，只需增加通道、重新训练，程序立刻变得更好。
Modules can meld into an optimal whole. Our software is often decomposed into modules that communicate through public functions, APIs, or endpoints.
模块能熔合成最优整体。我们的软件通常拆成模块，靠公开函数、API 或端点相互通信。
However, if two Software 2.0 modules that were originally trained separately interact, we can easily backpropagate through the whole.
但如果两个原本分开训练的软件 2.0 模块发生交互，我们可以轻松地对整体做反向传播。
Think about how amazing it could be if your web browser could automatically re-design the low-level system instructions 10 stacks down to achieve a higher efficiency in loading web pages.
想想这有多惊人：你的浏览器能自动重新设计十层栈之下的底层系统指令，来提高网页加载效率。
Or if the computer vision library (e.g. OpenCV) you imported could be auto-tuned on your specific data.
或者你引入的计算机视觉库（如 OpenCV）能在你的特定数据上自动调优。
With 2.0, this is the default behavior.
在 2.0 里，这是默认行为。
It is better than you. Finally, and most importantly, a neural network is a better piece of code than anything you or I can come up with in a large fraction of valuable verticals, which currently at the very least involve anything to do with images/video and sound/speech.
它比你写得好。最后也最重要：在一大批有价值的垂直领域里，神经网络就是比你我能写出的任何东西都更好的代码——目前至少涵盖一切与图像/视频、声音/语音有关的领域。
The 2.0 stack also has some of its own disadvantages.
2.0 技术栈也有它自己的缺点。
At the end of the optimization we're left with large networks that work well, but it's very hard to tell how.
优化结束后，我们得到的是效果很好的大网络，但很难说清它为什么好。
Across many applications areas, we'll be left with a choice of using a 90% accurate model we understand, or 99% accurate model we don't.
在许多应用领域，我们将面临这样的选择：用一个我们理解的 90% 准确率模型，还是一个我们不理解的 99% 准确率模型。
The 2.0 stack can fail in unintuitive and embarrassing ways, or worse, they can "silently fail", e.g., by silently adopting biases in their training data, which are very difficult to properly analyze and examine when their sizes are easily in the millions in most cases.
2.0 技术栈会以反直觉、令人难堪的方式失败，更糟的是还会"静默失败"——比如不声不响地吸收训练数据里的偏见；而当数据规模动辄数百万时，这些偏见极难被正经地分析和审查。
Finally, we're still discovering some of the peculiar properties of this stack.
最后，我们仍在陆续发现这套技术栈的一些古怪性质。
For instance, the existence of adversarial examples and attacks highlights the unintuitive nature of this stack.
比如，对抗样本（adversarial examples）与对抗攻击的存在，就凸显了它反直觉的本性。
Software 1.0 is code we write. Software 2.0 is code written by the optimization based on an evaluation criterion (such as "classify this training data correctly").
软件 1.0 是我们写的代码。软件 2.0 是优化过程依据某个评价标准（比如"把这批训练数据分类正确"）写出的代码。
It is likely that any setting where the program is not obvious but one can repeatedly evaluate the performance of it (e.g. — did you classify some images correctly? do you win games of Go?) will be subject to this transition, because the optimization can find much better code than what a human can write.
凡是程序本身不显而易见、但其表现可以被反复评估的场景（图像分类对了吗？围棋赢了吗？），大概率都会经历这场迁移——因为优化能找到远比人类写得出的更好的代码。
The lens through which we view trends matters.
我们观察趋势所用的透镜很重要。
If you recognize Software 2.0 as a new and emerging programming paradigm instead of simply treating neural networks as a pretty good classifier in the class of machine learning techniques, the extrapolations become more obvious, and it's clear that there is much more work to do.
如果你把软件 2.0 认作一种新兴的编程范式，而不是把神经网络仅仅当成机器学习技术里一个不错的分类器，那么外推的方向就清晰得多，而且显然还有大量工作要做。
In particular, we've built up a vast amount of tooling that assists humans in writing 1.0 code, such as powerful IDEs with features like syntax highlighting, debuggers, profilers, go to def, git integration, etc.
具体说，我们为帮助人类写 1.0 代码建起了海量工具：功能强大的 IDE，带语法高亮、调试器、性能分析器、跳转定义、git 集成等等。
In the 2.0 stack, the programming is done by accumulating, massaging and cleaning datasets.
而在 2.0 技术栈里，编程是靠积累、揉捏和清洗数据集完成的。
For example, when the network fails in some hard or rare cases, we do not fix those predictions by writing code, but by including more labeled examples of those cases.
比如，当网络在某些困难或罕见的案例上出错时，我们不是靠写代码去修正预测，而是把更多这类案例的标注样本收进数据集。
Who is going to develop the first Software 2.0 IDEs, which help with all of the workflows in accumulating, visualizing, cleaning, labeling, and sourcing datasets?
谁会开发出第一批软件 2.0 的 IDE，来支撑数据集的积累、可视化、清洗、标注与采集的全套工作流？
Perhaps the IDE bubbles up images that the network suspects are mislabeled based on the per-example loss, or assists in labeling by seeding labels with predictions, or suggests useful examples to label based on the uncertainty of the network's predictions.
也许这种 IDE 会根据逐样本损失把网络怀疑标错的图片顶上来，或者用模型预测预填标签来辅助标注，或者根据网络预测的不确定性建议哪些样本最值得去标。
Similarly, Github is a very successful home for Software 1.0 code. Is there space for a Software 2.0 Github?
同样，GitHub 是软件 1.0 代码极成功的家园。会不会有软件 2.0 的 GitHub？
In this case repositories are datasets and commits are made up of additions and edits of the labels.
在那里，仓库是数据集，提交由标签的增补与修改构成。
Traditional package managers and related serving infrastructure like pip, conda, docker, etc. help us more easily deploy and compose binaries.
传统的包管理器和相关部署设施——pip、conda、docker 等——帮我们更容易地部署和组合二进制程序。
How do we effectively deploy, share, import and work with Software 2.0 binaries? What is the conda equivalent for neural networks?
我们要怎样有效地部署、分享、引入和使用软件 2.0 的二进制产物？神经网络界的 conda 是什么？
In the short term, Software 2.0 will become increasingly prevalent in any domain where repeated evaluation is possible and cheap, and where the algorithm itself is difficult to design explicitly.
短期看，凡是能廉价地反复评估、而算法本身又难以显式设计的领域，软件 2.0 都会日益盛行。
There are many exciting opportunities to consider the entire software development ecosystem and how it can be adapted to this new programming paradigm.
把整个软件开发生态如何适配这一新范式想一遍，其中有许多激动人心的机会。
And in the long run, the future of this paradigm is bright because it is increasingly clear that when we develop AGI, it will certainly be written in Software 2.0.
而长期看，这一范式的前景光明——因为越来越清楚的是：当我们造出 AGI 时，它必定是用软件 2.0 写成的。