array(2) { ["docs"]=> array(10) { [0]=> array(10) { ["id"]=> string(3) "428" ["text"]=> string(77) "Visual Studio 2017 单独启动MSDN帮助（Microsoft Help Viewer）的方法" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(8) "DonetRen" ["tagsname"]=> string(55) "Visual Studio 2017|MSDN帮助|C#程序|.NET|Help Viewer" ["tagsid"]=> string(23) "[401,402,403,"300",404]" ["catesname"]=> string(0) "" ["catesid"]=> string(2) "[]" ["createtime"]=> string(10) "1511400964" ["_id"]=> string(3) "428" } [1]=> array(10) { ["id"]=> string(3) "427" ["text"]=> string(42) "npm -v；报错 cannot find module "wrapp"" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(4) "zzty" ["tagsname"]=> string(50) "node.js|npm|cannot find module "wrapp“|node" ["tagsid"]=> string(19) "[398,"239",399,400]" ["catesname"]=> string(0) "" ["catesid"]=> string(2) "[]" ["createtime"]=> string(10) "1511400760" ["_id"]=> string(3) "427" } [2]=> array(10) { ["id"]=> string(3) "426" ["text"]=> string(54) "说说css中pt、px、em、rem都扮演了什么角色" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(12) "zhengqiaoyin" ["tagsname"]=> string(0) "" ["tagsid"]=> string(2) "[]" ["catesname"]=> string(0) "" ["catesid"]=> string(2) "[]" ["createtime"]=> string(10) "1511400640" ["_id"]=> string(3) "426" } [3]=> array(10) { ["id"]=> string(3) "425" ["text"]=> string(83) "深入学习JS执行--创建执行上下文（变量对象，作用域链，this）" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(7) "Ry-yuan" ["tagsname"]=> string(33) "Javascript|Javascript执行过程" ["tagsid"]=> string(13) "["169","191"]" ["catesname"]=> string(0) "" ["catesid"]=> string(2) "[]" ["createtime"]=> string(10) "1511399901" ["_id"]=> string(3) "425" } [4]=> array(10) { ["id"]=> string(3) "424" ["text"]=> string(30) "C# 排序技术研究与对比" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(9) "vveiliang" ["tagsname"]=> string(0) "" ["tagsid"]=> string(2) "[]" ["catesname"]=> string(8) ".Net Dev" ["catesid"]=> string(5) "[199]" ["createtime"]=> string(10) "1511399150" ["_id"]=> string(3) "424" } [5]=> array(10) { ["id"]=> string(3) "423" ["text"]=> string(72) "【算法】小白的算法笔记：快速排序算法的编码和优化" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(9) "penghuwan" ["tagsname"]=> string(6) "算法" ["tagsid"]=> string(7) "["344"]" ["catesname"]=> string(0) "" ["catesid"]=> string(2) "[]" ["createtime"]=> string(10) "1511398109" ["_id"]=> string(3) "423" } [6]=> array(10) { ["id"]=> string(3) "422" ["text"]=> string(64) "JavaScript数据可视化编程学习（二）Flotr2，雷达图" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(7) "chengxs" ["tagsname"]=> string(28) "数据可视化|前端学习" ["tagsid"]=> string(9) "[396,397]" ["catesname"]=> string(18) "前端基本知识" ["catesid"]=> string(5) "[198]" ["createtime"]=> string(10) "1511397800" ["_id"]=> string(3) "422" } [7]=> array(10) { ["id"]=> string(3) "421" ["text"]=> string(36) "C#表达式目录树（Expression）" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(4) "wwym" ["tagsname"]=> string(0) "" ["tagsid"]=> string(2) "[]" ["catesname"]=> string(4) ".NET" ["catesid"]=> string(7) "["119"]" ["createtime"]=> string(10) "1511397474" ["_id"]=> string(3) "421" } [8]=> array(10) { ["id"]=> string(3) "420" ["text"]=> string(47) "数据结构 队列_队列实例：事件处理" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(7) "idreamo" ["tagsname"]=> string(40) "C语言|数据结构|队列|事件处理" ["tagsid"]=> string(23) "["246","247","248",395]" ["catesname"]=> string(12) "数据结构" ["catesid"]=> string(7) "["133"]" ["createtime"]=> string(10) "1511397279" ["_id"]=> string(3) "420" } [9]=> array(10) { ["id"]=> string(3) "419" ["text"]=> string(47) "久等了,博客园官方Android客户端发布" ["intro"]=> string(288) "目录 ECharts 异步加载 ECharts 数据可视化在过去几年中取得了巨大进展。开发人员对可视化产品的期望不再是简单的图表创建工具，而是在交互、性能、数据处理等方面有更高的要求。 chart.setOption({ color: [ " ["username"]=> string(3) "cmt" ["tagsname"]=> string(0) "" ["tagsid"]=> string(2) "[]" ["catesname"]=> string(0) "" ["catesid"]=> string(2) "[]" ["createtime"]=> string(10) "1511396549" ["_id"]=> string(3) "419" } } ["count"]=> int(200) } 222 CMU 11-785 L02 What can a network represent - 爱码网

CMU 11-785 L02 What can a network represent

Preliminary

Perceptron

  • Threshold unit
    • “Fires” if the weighted sum of inputs exceeds a threshold
  • Soft perceptron
    • Using sigmoid function instead of a threshold at the output
    • Activation: The function that acts on the weighted combination of inputs (and threshold)
  • Affine combination
    • Different from Linear combination: the result of mapping zero is not zero.

Multi-layer perceptron

  • Depth
    • Is the length of the longest path from a source to a sink
    • Deep: Depth greater than 2
  • Inputs/Outputs are real or Boolean stimuli
  • What can this network compute?

Universal Boolean functions

  • A perceptron can model any simple binary Boolean gate
    • Using weight 1 or -1 to model function
    • The universal AND gate: (⋀i=1LXi)∧(⋀i=L+1NXˉi)(\bigwedge_{i=1}^{L} X_{i}) \wedge(\bigwedge_{i=L+1}^{N} \bar{X}_{i})
    • The universal OR gate: (⋁i=1LXi)∨(⋁i=L+1NXˉi)(\bigvee_{i=1}^{L} X_{i}) \vee(\bigvee_{i=L+1}^{N} \bar{X}_{i})
    • Cannot compute an XOR
  • MLPs can compute the XOR

CMU 11-785 L02 What can a network represent

  • MLPs are universal Boolean functions

    • Can compute any Boolean function

  • A Boolean function is just a truth table

    • So expressed the result in disjunctive normal form, like

    • Y=Xˉ1Xˉ2X3X4Xˉ5+Xˉ1X2Xˉ3X4X5+Xˉ1X2X3Xˉ4Xˉ5+X1Xˉ2Xˉ3Xˉ4X5+X1Xˉ2X3X4X5+X1X2Xˉ3Xˉ4X5 \begin{aligned} Y=& \bar{X}_{1} \bar{X}_{2} X_{3} X_{4} \bar{X}_{5}+\bar{X}_{1} X_{2} \bar{X}_{3} X_{4} X_{5}+\bar{X}_{1} X_{2} X_{3} \bar{X}_{4} \bar{X}_{5}+\\ & X_{1} \bar{X}_{2} \bar{X}_{3} \bar{X}_{4} X_{5}+X_{1} \bar{X}_{2} X_{3} X_{4} X_{5}+X_{1} X_{2} \bar{X}_{3} \bar{X}_{4} X_{5} \end{aligned}

    • In this case, need 5 neurons in the hidden layer.

Need for depth

  • A one-hidden-layer MLP is a Universal Boolean Function

    • But the largest number of perceptrons is expontial: 2N2^N

  • How about depth?

    • Will require 3(N−1)3(N-1) perceptrons, linear in NN to express the same function
    • Using associatable rules, can be arranged in 2log⁡2N2\log_2 N layers
    • eg. model O=W⊕X⊕Y⊕ZO=W \oplus X \oplus Y \oplus Z

CMU 11-785 L02 What can a network represent

  • The challenge of depth

    • Using only KK hidden layers will require O(2CN)O(2^{CN}) neurons in the KKth layer, where C=2−(k−1)/2C = 2^{-(k-1)/2}
    • A network with fewer than the minimum required number of neurons cannot model the function

Universal classifiers

  • Composing complicated “decision” boundaries

CMU 11-785 L02 What can a network represent

  • Using OR to create more decision boundaries
    • Can compose arbitrarily complex decision boundaries
    • Even using one-layer MLP

Need for depth

  • A naïve one-hidden-layer neural network will required infinite hidden neurons
  • Construct basic unit and add more layers to decrese #neurons
  • The number of neurons required in a shallow network is potentially exponential in the dimensionality of the input

Universal approximators

  • A one-layer MLP can model an arbitrary function of a single input
  • MLPs can actually compose arbitrary functions in any number of dimensions
    • Even without “activation”
  • Activation
    • A universal map from the entire domain of input values to the entire range of the output activation

Optimal depth and width

  • Deeper networks will require far fewer neurons for the same approximation error
  • Sufficiency of architecture
    • Not all architectures can represent any function
  • Continuous activation functions result in graded output at the layer
    • To capture information “missed” by the lower layer

Width vs. Activations vs. Depth

  • Narrow layers can still pass information to subsequent layers if the activation function is sufficiently graded
    • But will require greater depth, to permit later layers to capture patterns
  • Capacity of the network
    • Information or Storage: how many patterns can it remember
    • VC dimension: bounded by the square of the number of …weights… in the network
    • Straight forward: largest number of disconnected convex regions it can represent
  • A network with insufficient capacity cannot exactly model a function that requires a greater minimal number of convex hulls than the capacity of the network

相关文章：

猜你喜欢
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode