-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy path2019011401.html
More file actions
1 lines (1 loc) · 46.7 KB
/
Copy path2019011401.html
File metadata and controls
1 lines (1 loc) · 46.7 KB
1
<!DOCTYPE html><html class="theme-next mist use-motion" lang="zh-Hans"><head><meta name="generator" content="Hexo 3.9.0"><meta charset="UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1"><meta name="theme-color" content="#222"><script src="/lib/pace/pace.min.js?v=1.0.2"></script><link href="/lib/pace/pace-theme-minimal.min.css?v=1.0.2" rel="stylesheet"><meta http-equiv="Cache-Control" content="no-transform"><meta http-equiv="Cache-Control" content="no-siteapp"><link href="/lib/fancybox/source/jquery.fancybox.css?v=2.1.5" rel="stylesheet" type="text/css"><link href="/lib/font-awesome/css/font-awesome.min.css?v=4.6.2" rel="stylesheet" type="text/css"><link href="/css/main.css?v=5.1.3" rel="stylesheet" type="text/css"><link rel="apple-touch-icon" sizes="180x180" href="/images/apple-touch-icon-240x240-playpi.png?v=5.1.3"><link rel="icon" type="image/png" sizes="32x32" href="/images/favicon-32x32-playpi.png?v=5.1.3"><link rel="icon" type="image/png" sizes="16x16" href="/images/favicon-16x16-playpi.png?v=5.1.3"><link rel="mask-icon" href="/images/logo-playpi.svg?v=5.1.3" color="#222"><meta name="keywords" content="Spark,netty,nio"><link rel="alternate" href="/atom.xml" title="虾丸派" type="application/atom+xml"><meta name="description" content="在做项目的时候，需要新引入一个外部依赖，于是很自然地在项目的 pom.xml 文件中加入了依赖坐标，然后进行编译、打包、运行，没想到直接抛出了异常：122019-01-13_17:18:52 [sparkDriverActorSystem-akka.actor.default-dispatcher-5] ERROR actor.ActorSystemImpl:66: Uncaught fatal"><meta name="keywords" content="Spark,netty,nio"><meta property="og:type" content="article"><meta property="og:title" content="Spark 异常之 Netty 相关"><meta property="og:url" content="https://www.playpi.org/2019011401.html"><meta property="og:site_name" content="虾丸派"><meta property="og:description" content="在做项目的时候，需要新引入一个外部依赖，于是很自然地在项目的 pom.xml 文件中加入了依赖坐标，然后进行编译、打包、运行，没想到直接抛出了异常：122019-01-13_17:18:52 [sparkDriverActorSystem-akka.actor.default-dispatcher-5] ERROR actor.ActorSystemImpl:66: Uncaught fatal"><meta property="og:locale" content="zh-Hans"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j8uvmnrj219y0kqdil.jpg"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j9dcym1j216q0aztai.jpg"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j9wfx1mj20wu0et3z8.jpg"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jagdz1qj216t09wwg0.jpg"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jaxljmvj20mb05fglv.jpg"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jb7z4o0j20la031t8r.jpg"><meta property="og:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jbj046wj20g907amxa.jpg"><meta property="og:updated_time" content="2019-01-13T17:27:53.000Z"><meta name="twitter:card" content="summary"><meta name="twitter:title" content="Spark 异常之 Netty 相关"><meta name="twitter:description" content="在做项目的时候，需要新引入一个外部依赖，于是很自然地在项目的 pom.xml 文件中加入了依赖坐标，然后进行编译、打包、运行，没想到直接抛出了异常：122019-01-13_17:18:52 [sparkDriverActorSystem-akka.actor.default-dispatcher-5] ERROR actor.ActorSystemImpl:66: Uncaught fatal"><meta name="twitter:image" content="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j8uvmnrj219y0kqdil.jpg"><script type="text/javascript" id="hexo.configurations">var NexT=window.NexT||{},CONFIG={root:"/",scheme:"Mist",version:"5.1.3",sidebar:{position:"left",display:"hide",offset:12,b2t:!1,scrollpercent:!0,onmobile:!1},fancybox:!0,tabs:!0,motion:{enable:!0,async:!1,transition:{post_block:"fadeIn",post_header:"slideDownIn",post_body:"slideDownIn",coll_header:"slideLeftIn",sidebar:"slideUpIn"}},duoshuo:{userId:"0",author:"博主"},algolia:{applicationID:"",apiKey:"",indexName:"",hits:{per_page:10},labels:{input_placeholder:"Search for Posts",hits_empty:"We didn't find any results for the search: ${query}",hits_stats:"${hits} results found in ${time} ms"}}}</script><link rel="canonical" href="https://www.playpi.org/2019011401.html"><title>Spark 异常之 Netty 相关 | 虾丸派</title></head><body itemscope itemtype="http://schema.org/WebPage" lang="zh-Hans"><div class="container sidebar-position-left page-post-detail"><div class="headband"></div><header id="header" class="header" itemscope itemtype="http://schema.org/WPHeader"><div class="header-inner"><div class="site-brand-wrapper"><div class="site-meta"><div class="custom-logo-site-title"><a href="/" class="brand" rel="start"><span class="logo-line-before"><i></i></span> <span class="site-title">虾丸派</span> <span class="logo-line-after"><i></i></span></a></div><h1 class="site-subtitle" itemprop="description">烂笔头</h1></div><div class="site-nav-toggle"><button><span class="btn-bar"></span> <span class="btn-bar"></span> <span class="btn-bar"></span></button></div></div><nav class="site-nav"><ul id="menu" class="menu"><li class="menu-item menu-item-home"><a href="/" rel="section"><i class="menu-item-icon fa fa-fw fa-home"></i><br>首页</a></li><li class="menu-item menu-item-tags"><a href="/tags/" rel="section"><i class="menu-item-icon fa fa-fw fa-tags"></i><br>标签</a></li><li class="menu-item menu-item-categories"><a href="/categories/" rel="section"><i class="menu-item-icon fa fa-fw fa-th"></i><br>分类</a></li><li class="menu-item menu-item-archives"><a href="/archives/" rel="section"><i class="menu-item-icon fa fa-fw fa-archive"></i><br>归档</a></li><li class="menu-item menu-item-about"><a href="/about/" rel="section"><i class="menu-item-icon fa fa-fw fa-user"></i><br>关于</a></li><li class="menu-item menu-item-books"><a href="/books/" rel="section"><i class="menu-item-icon fa fa-fw fa-book"></i><br>书籍</a></li><li class="menu-item menu-item-guide"><a href="/guide/" rel="section"><i class="menu-item-icon fa fa-fw fa-location-arrow"></i><br>指南</a></li><li class="menu-item menu-item-search"><a href="javascript:;" class="popup-trigger"><i class="menu-item-icon fa fa-search fa-fw"></i><br>搜索</a></li></ul><div class="site-search"><div class="popup search-popup local-search-popup"><div class="local-search-header clearfix"><span class="search-icon"><i class="fa fa-search"></i> </span><span class="popup-btn-close"><i class="fa fa-times-circle"></i></span><div class="local-search-input-wrapper"><input autocomplete="off" placeholder="搜索..." spellcheck="false" type="text" id="local-search-input"></div></div><div id="local-search-result"></div></div></div></nav></div></header><main id="main" class="main"><div class="main-inner"><div class="content-wrap"><div id="content" class="content"><div id="posts" class="posts-expand"><article class="post post-type-normal" itemscope itemtype="http://schema.org/Article"><div class="post-block"><link itemprop="mainEntityOfPage" href="https://www.playpi.org/2019011401.html"><span hidden itemprop="author" itemscope itemtype="http://schema.org/Person"><meta itemprop="name" content="虾丸派"><meta itemprop="description" content="记录知识 | 分享技术"><meta itemprop="image" content="/images/favicon-1536x1536-playpi.png"></span><span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization"><meta itemprop="name" content="虾丸派"></span><header class="post-header"><h2 class="post-title" itemprop="name headline">Spark 异常之 Netty 相关</h2><div class="post-meta"><span class="post-time"><span class="post-meta-item-text">发表于</span> <time title="创建于" itemprop="dateCreated datePublished" datetime="2019-01-14T01:27:53+08:00">2019-01-14 </time></span><span class="post-category"><span class="post-meta-divider">|</span> <span class="post-meta-item-text">分类于</span> <span itemprop="about" itemscope itemtype="http://schema.org/Thing"><a href="/categories/basic-technical-knowledge/" itemprop="url" rel="index"><span itemprop="name">基础技术知识</span> </a></span></span><span id="busuanzi_container_page_pv" style="display:none"><span class="post-meta-divider">|</span> 阅读次数 <span id="busuanzi_value_page_pv"></span></span><div class="post-wordcount"><span class="post-meta-item-text">字数统计</span> <span title="字数统计">2,017字 </span><span class="post-meta-divider">|</span> <span class="post-meta-item-text">阅读时长 ≈</span> <span title="阅读时长">10分钟</span></div></div></header><div class="post-body" itemprop="articleBody"><p>在做项目的时候，需要新引入一个外部依赖，于是很自然地在项目的 pom.xml 文件中加入了依赖坐标，然后进行编译、打包、运行，没想到直接抛出了异常：</p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">18</span>:<span class="number">52</span> [sparkDriverActorSystem-akka.actor.<span class="keyword">default</span>-dispatcher-<span class="number">5</span>] ERROR actor.ActorSystemImpl:<span class="number">66</span>: Uncaught fatal error from thread [sparkDriverActorSystem-akka.remote.<span class="keyword">default</span>-remote-dispatcher-<span class="number">7</span>] shutting down ActorSystem [sparkDriverActorSystem]</span><br><span class="line">java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;) Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function</span><br></pre></td></tr></table></figure><p>任务运行失败，仔细看日志觉得很莫名奇妙，是一个 java.lang.VerifyError 错误，以前从来没见过类似的。本文记录这个错误的解决过程。</p><a id="more"></a><h1 id="问题出现"><a href="# 问题出现" class="headerlink" title="问题出现"></a>问题出现</h1><p>在上述错误抛出之后，可以看到 SparkContext 初始化失败，然后进程就终止了；</p><p>完整日志如下：<br></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">18</span>:<span class="number">52</span> [sparkDriverActorSystem-akka.actor.<span class="keyword">default</span>-dispatcher-<span class="number">5</span>] ERROR actor.ActorSystemImpl:<span class="number">66</span>: Uncaught fatal error from thread [sparkDriverActorSystem-akka.remote.<span class="keyword">default</span>-remote-dispatcher-<span class="number">7</span>] shutting down ActorSystem [sparkDriverActorSystem]</span><br><span class="line">java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;) Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function</span><br><span class="line"> at akka.remote.transport.netty.NettyTransport.(NettyTransport.scala:<span class="number">283</span>)</span><br><span class="line"> at akka.remote.transport.netty.NettyTransport.(NettyTransport.scala:<span class="number">240</span>)</span><br><span class="line"> at sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method)</span><br><span class="line"> at sun.reflect.NativeConstructorAccessorImpl.newInstance (NativeConstructorAccessorImpl.java:<span class="number">62</span>)</span><br><span class="line"> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance (DelegatingConstructorAccessorImpl.java:<span class="number">45</span>)</span><br><span class="line"> at java.lang.reflect.Constructor.newInstance (Constructor.java:<span class="number">423</span>)</span><br><span class="line"> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$<span class="number">2</span>.apply (DynamicAccess.scala:<span class="number">78</span>)</span><br><span class="line"> at scala.util.Try$.apply (Try.scala:<span class="number">161</span>)</span><br><span class="line"> at akka.actor.ReflectiveDynamicAccess.createInstanceFor (DynamicAccess.scala:<span class="number">73</span>)</span><br><span class="line"> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$<span class="number">3</span>.apply (DynamicAccess.scala:<span class="number">84</span>)</span><br><span class="line"> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$<span class="number">3</span>.apply (DynamicAccess.scala:<span class="number">84</span>)</span><br><span class="line"> at scala.util.Success.flatMap (Try.scala:<span class="number">200</span>)</span><br><span class="line"> at akka.actor.ReflectiveDynamicAccess.createInstanceFor (DynamicAccess.scala:<span class="number">84</span>)</span><br><span class="line"> at akka.remote.EndpointManager$$anonfun$<span class="number">9</span>.apply (Remoting.scala:<span class="number">711</span>)</span><br><span class="line"> at akka.remote.EndpointManager$$anonfun$<span class="number">9</span>.apply (Remoting.scala:<span class="number">703</span>)</span><br><span class="line"> at scala.collection.TraversableLike$WithFilter$$anonfun$map$<span class="number">2</span>.apply (TraversableLike.scala:<span class="number">722</span>)</span><br><span class="line"> at scala.collection.Iterator$class.foreach (Iterator.scala:727)</span><br><span class="line"> at scala.collection.AbstractIterator.foreach (Iterator.scala:<span class="number">1157</span>)</span><br><span class="line"> at scala.collection.IterableLike$class.foreach (IterableLike.scala:72)</span><br><span class="line"> at scala.collection.AbstractIterable.foreach (Iterable.scala:<span class="number">54</span>)</span><br><span class="line"> at scala.collection.TraversableLike$WithFilter.map (TraversableLike.scala:<span class="number">721</span>)</span><br><span class="line"> at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens (Remoting.scala:<span class="number">703</span>)</span><br><span class="line"> at akka.remote.EndpointManager$$anonfun$receive$<span class="number">2</span>.applyOrElse (Remoting.scala:<span class="number">491</span>)</span><br><span class="line"> at akka.actor.Actor$class.aroundReceive (Actor.scala:467)</span><br><span class="line"> at akka.remote.EndpointManager.aroundReceive (Remoting.scala:<span class="number">394</span>)</span><br><span class="line"> at akka.actor.ActorCell.receiveMessage (ActorCell.scala:<span class="number">516</span>)</span><br><span class="line"> at akka.actor.ActorCell.invoke (ActorCell.scala:<span class="number">487</span>)</span><br><span class="line"> at akka.dispatch.Mailbox.processMailbox (Mailbox.scala:<span class="number">238</span>)</span><br><span class="line"> at akka.dispatch.Mailbox.run (Mailbox.scala:<span class="number">220</span>)</span><br><span class="line"> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec (AbstractDispatcher.scala:<span class="number">397</span>)</span><br><span class="line"> at scala.concurrent.forkjoin.ForkJoinTask.doExec (ForkJoinTask.java:<span class="number">260</span>)</span><br><span class="line"> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask (ForkJoinPool.java:<span class="number">1339</span>)</span><br><span class="line"> at scala.concurrent.forkjoin.ForkJoinPool.runWorker (ForkJoinPool.java:<span class="number">1979</span>)</span><br><span class="line"> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run (ForkJoinWorkerThread.java:<span class="number">107</span>)</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">18</span>:<span class="number">52</span> [sparkDriverActorSystem-akka.actor.<span class="keyword">default</span>-dispatcher-<span class="number">6</span>] INFO remote.RemoteActorRefProvider$RemotingTerminator:<span class="number">74</span>: Shutting down remote daemon.</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">18</span>:<span class="number">52</span> [sparkDriverActorSystem-akka.actor.<span class="keyword">default</span>-dispatcher-<span class="number">6</span>] INFO remote.RemoteActorRefProvider$RemotingTerminator:<span class="number">74</span>: Remote daemon shut down; proceeding with flushing remote transports.</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">18</span>:<span class="number">52</span> [sparkDriverActorSystem-akka.actor.<span class="keyword">default</span>-dispatcher-<span class="number">6</span>] ERROR Remoting:<span class="number">65</span>: Remoting system has been terminated abrubtly. Attempting to shut down transports</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">18</span>:<span class="number">52</span> [sparkDriverActorSystem-akka.actor.<span class="keyword">default</span>-dispatcher-<span class="number">6</span>] INFO remote.RemoteActorRefProvider$RemotingTerminator:<span class="number">74</span>: Remoting shut down.</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">19</span>:<span class="number">02</span> [main] ERROR spark.SparkContext:<span class="number">95</span>: Error initializing SparkContext.</span><br><span class="line">java.util.concurrent.TimeoutException: Futures timed out after [<span class="number">10000</span> milliseconds]</span><br><span class="line"> at scala.concurrent.impl.Promise$DefaultPromise.ready (Promise.scala:<span class="number">219</span>)</span><br><span class="line"> at scala.concurrent.impl.Promise$DefaultPromise.result (Promise.scala:<span class="number">223</span>)</span><br><span class="line"> at scala.concurrent.Await$$anonfun$result$<span class="number">1</span>.apply (<span class="keyword">package</span>.scala:<span class="number">107</span>)</span><br><span class="line"> at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn (BlockContext.scala:<span class="number">53</span>)</span><br><span class="line"> at scala.concurrent.Await$.result (<span class="keyword">package</span>.scala:<span class="number">107</span>)</span><br><span class="line"> at akka.remote.Remoting.start (Remoting.scala:<span class="number">179</span>)</span><br><span class="line"> at akka.remote.RemoteActorRefProvider.init (RemoteActorRefProvider.scala:<span class="number">184</span>)</span><br><span class="line"> at akka.actor.ActorSystemImpl.liftedTree2$<span class="number">1</span>(ActorSystem.scala:<span class="number">620</span>)</span><br><span class="line"> at akka.actor.ActorSystemImpl._start$lzycompute (ActorSystem.scala:<span class="number">617</span>)</span><br><span class="line"> at akka.actor.ActorSystemImpl._start (ActorSystem.scala:<span class="number">617</span>)</span><br><span class="line"> at akka.actor.ActorSystemImpl.start (ActorSystem.scala:<span class="number">634</span>)</span><br><span class="line"> at akka.actor.ActorSystem$.apply (ActorSystem.scala:<span class="number">142</span>)</span><br><span class="line"> at akka.actor.ActorSystem$.apply (ActorSystem.scala:<span class="number">119</span>)</span><br><span class="line"> at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem (AkkaUtils.scala:<span class="number">121</span>)</span><br><span class="line"> at org.apache.spark.util.AkkaUtils$$anonfun$<span class="number">1</span>.apply (AkkaUtils.scala:<span class="number">53</span>)</span><br><span class="line"> at org.apache.spark.util.AkkaUtils$$anonfun$<span class="number">1</span>.apply (AkkaUtils.scala:<span class="number">52</span>)</span><br><span class="line"> at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$<span class="number">1</span>.apply$mcVI$sp (Utils.scala:<span class="number">2024</span>)</span><br><span class="line"> at scala.collection.immutable.Range.foreach$mVc$sp (Range.scala:<span class="number">141</span>)</span><br><span class="line"> at org.apache.spark.util.Utils$.startServiceOnPort (Utils.scala:<span class="number">2015</span>)</span><br><span class="line"> at org.apache.spark.util.AkkaUtils$.createActorSystem (AkkaUtils.scala:<span class="number">55</span>)</span><br><span class="line"> at org.apache.spark.SparkEnv$.create (SparkEnv.scala:<span class="number">266</span>)</span><br><span class="line"> at org.apache.spark.SparkEnv$.createDriverEnv (SparkEnv.scala:<span class="number">193</span>)</span><br><span class="line"> at org.apache.spark.SparkContext.createSparkEnv (SparkContext.scala:<span class="number">288</span>)</span><br><span class="line"> at org.apache.spark.SparkContext.(SparkContext.scala:<span class="number">457</span>)</span><br><span class="line"> at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:<span class="number">59</span>)</span><br><span class="line"> at com.ds.octopus.job.utils.SparkContextUtil.refresh (SparkContextUtil.java:<span class="number">77</span>)</span><br><span class="line"> at com.ds.octopus.job.utils.SparkContextUtil.getJsc (SparkContextUtil.java:<span class="number">34</span>)</span><br><span class="line"> at com.ds.octopus.job.executors.impl.WeiboZPZExporter.action (WeiboZPZExporter.java:<span class="number">95</span>)</span><br><span class="line"> at com.ds.octopus.job.executors.impl.WeiboZPZExporter.action (WeiboZPZExporter.java:<span class="number">41</span>)</span><br><span class="line"> at com.ds.octopus.job.executors.SimpleExecutor.execute (SimpleExecutor.java:<span class="number">40</span>)</span><br><span class="line"> at com.ds.octopus.job.client.OctopusClient.run (OctopusClient.java:<span class="number">162</span>)</span><br><span class="line"> at com.yeezhao.commons.buffalo.job.AbstractBUTaskWorker.runTask (AbstractBUTaskWorker.java:<span class="number">63</span>)</span><br><span class="line"> at com.ds.octopus.job.client.TaskLocalRunnerCli.start (TaskLocalRunnerCli.java:<span class="number">109</span>)</span><br><span class="line"> at com.yeezhao.commons.util.AdvCli.initRunner (AdvCli.java:<span class="number">191</span>)</span><br><span class="line"> at com.ds.octopus.job.client.TaskLocalRunnerCli.main (TaskLocalRunnerCli.java:<span class="number">41</span>)</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">13_17</span>:<span class="number">19</span>:<span class="number">02</span> [main] INFO spark.SparkContext:<span class="number">58</span>: Successfully stopped SparkContext</span><br></pre></td></tr></table></figure><p></p><p>错误日志截图：<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j8uvmnrj219y0kqdil.jpg" alt="错误日志局部" title="错误日志局部"></p><p>根据日志没有看出有关 Java 层面的什么问题，只能根据 JNI 字段描述符：<br></p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">class: org/jboss/netty/channel/socket/nio/NioWorkerPool</span><br></pre></td></tr></table></figure><p></p><p>猜测是某一个类的问题，根据：<br></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">method: createWorker signature: (Ljava/util/concurrent/Executor;) Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong <span class="keyword">return</span> type in function</span><br></pre></td></tr></table></figure><p></p><p>猜测是某个方法的问题，方法的返回类型错误。</p><p>然后在项目中使用 ctrl+shift+t 快捷键（全局搜索 Java 类，每个人的开发工具设置的可能不一样）搜索类：NioWorkerPool，发现这个类的来源不是新引入的依赖包，而是原本就有的 netty 相关包，所以此时就可以断定这个莫名其妙的错误的原因就在于这个类的 createWorker 方法返回类型上面了。</p><p>搜索类 NioWorkerPool<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j9dcym1j216q0aztai.jpg" alt="搜索 NioWorkerPool" title="搜索 NioWorkerPool"></p><p>日志的 JNI 字段描述符显示返回类型是 AbstractNioWorker，但是这个一看就是抽象类，不是我们要找的，去类里面看源码，发现 createWorker 方法返回类型是 NioWorker：</p><p>类 NioWorkerPool 源码<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5j9wfx1mj20wu0et3z8.jpg" alt="NioWorkerPool 源码" title="NioWorkerPool 源码"></p><p>继续搜索类 NioWorker<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jagdz1qj216t09wwg0.jpg" alt="搜索 NioWorker" title="搜索 NioWorker"></p><p>好，此时发现问题了，这个类有 2 个，居然存在两个相同的包名，但是依赖坐标不一样，所以这个隐藏的原因在于类冲突，但是并不能算是依赖冲突引起的。也就是说，NioWorker 这个类重复了，但是依赖包坐标不一样，类的包路径却是一模一样的，不会引起版本冲突问题，而在实际运行任务的时候会抛出运行时异常，所以我觉得找问题的过程很艰辛。</p><p>使用依赖树查看依赖关系，是看不到版本冲突问题的，2 个依赖都存在：<br>io.netty 依赖<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jaxljmvj20mb05fglv.jpg" alt="io.netty 依赖" title="io.netty 依赖"></p><p>org.jboss.netty 依赖<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jb7z4o0j20la031t8r.jpg" alt="org.jboss.netty 依赖" title="org.jboss.netty 依赖"></p><p>于是又在网上搜索了一下，发现果然是 netty 的问题，也就是新引入的依赖包导致的，但是根本原因令人哭笑：netty 的组织结构变化，发布的依赖坐标名称变化，但是所有的类的包名称并没有变化，导致了这个错误。</p><h1 id="问题解决"><a href="# 问题解决" class="headerlink" title="问题解决"></a>问题解决</h1><p>问题找到了，解决方法就简单了，移除传递依赖即可，同时也要注意以后再添加新的依赖一定要慎重，不然找问题的过程很是令人崩溃。</p><p>移除依赖<br><img src="https://raw.githubusercontent.com/iplaypi/img-playpi/master/img/old/b7f2e3a3gy1fz5jbj046wj20g907amxa.jpg" alt="移除依赖" title="移除依赖"></p><p>移除配置示例<br></p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"><!-- 移除引发冲突的 jar 包 --></span></span><br><span class="line"><span class="tag"><<span class="name">exclusions</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclusion</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.jboss.netty<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>netty<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">exclusion</span>></span></span><br><span class="line"><span class="tag"></<span class="name">exclusions</span>></span></span><br></pre></td></tr></table></figure><p></p><h1 id="问题总结"><a href="# 问题总结" class="headerlink" title="问题总结"></a> 问题总结</h1><p>1、参考：<a href="https://stackoverflow.com/questions/33573587/apache-spark-wrong-akka-remote-netty-version" target="_blank" rel="noopener">https://stackoverflow.com/questions/33573587/apache-spark-wrong-akka-remote-netty-version</a> ；</p><p>2、netty 的组织结构（影响发布的 jar 包坐标名称）变化了，但是所有的类的包名称仍然是一致的，很奇怪，导致我找问题也觉得莫名其妙，因为这不会引发版本冲突问题（但是本质上又是 2 个一模一样的类被同时使用，引发类冲突）；</p><p>3、这个错误信息挺有意思的，解决过程也很好玩，边找边学习；</p><p>4、对于这种重名的类【类的包路径名、类名】，竟然对应的 jar 包不一样，这种极其特殊的情况也可以使用插件检测出来：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><groupId>org.apache.maven.plugins</groupId></span><br><span class="line"><artifactId>maven-enforcer-plugin</artifactId></span><br></pre></td></tr></table></figure><p>使用 <strong>enforcer:enforce</strong> 命令即可。</p><p>当然，这个插件还可以用来校验很多地方，例如代码中引用了 <strong>@Deprecated</strong> 的方法，也会给出提示信息，可以按照需求给插件配置需要校验的方面。</p></div><div><div id="wechat_subscriber" style="display:block;padding:10px 0;margin:20px auto;width:100%;text-align:center"><img id="wechat_subscriber_qcode" src="/images/wechat-qr-personal.jpg" alt="虾丸派 wechat" style="width:200px;max-width:100%"><div>扫一扫添加博主，进技术交流群，共同学习进步</div></div></div><div><div style="padding:10px 0;margin:20px auto;width:90%;text-align:center"><div>永不止步</div><button id="rewardButton" disable="enable" onclick='var qr=document.getElementById("QR");"none"===qr.style.display?qr.style.display="block":qr.style.display="none"'><span>打赏</span></button><div id="QR" style="display:none"><div id="wechat" style="display:inline-block"><img id="wechat_qr" src="/images/wechat-pay-playpi.png" alt="虾丸派 微信支付"><p>微信支付</p></div></div></div></div><div><ul class="post-copyright"><li class="post-copyright-author"><strong>本文作者：</strong> 虾丸派</li><li class="post-copyright-link"><strong>本文链接：</strong> <a href="https://www.playpi.org/2019011401.html" title="Spark 异常之 Netty 相关">https://www.playpi.org/2019011401.html</a></li><li class="post-copyright-license"><strong>版权声明： </strong>本博客所有文章除特别声明外，均采用 <a href="https://creativecommons.org/licenses/by-nc-sa/3.0/" rel="external nofollow" target="_blank">CC BY-NC-SA 3.0</a> 许可协议。转载请注明出处！</li></ul></div><footer class="post-footer"><div class="post-tags"><a href="/tags/Spark/" rel="tag"><i class="fa fa-tag"></i> Spark</a> <a href="/tags/netty/" rel="tag"><i class="fa fa-tag"></i> netty</a> <a href="/tags/nio/" rel="tag"><i class="fa fa-tag"></i> nio</a></div><div class="post-nav"><div class="post-nav-next post-nav-item"><a href="/2019011101.html" rel="next" title="Maven 插件异常之 socket write error"><i class="fa fa-chevron-left"></i> Maven 插件异常之 socket write error</a></div><span class="post-nav-divider"></span><div class="post-nav-prev post-nav-item"><a href="/2019020101.html" rel="prev" title="微博电影文稿备份 2">微博电影文稿备份 2 <i class="fa fa-chevron-right"></i></a></div></div></footer></div></article><div class="post-spread"></div></div></div><div class="comments" id="comments"><div id="vcomments"></div></div></div><div class="sidebar-toggle"><div class="sidebar-toggle-line-wrap"><span class="sidebar-toggle-line sidebar-toggle-line-first"></span> <span class="sidebar-toggle-line sidebar-toggle-line-middle"></span> <span class="sidebar-toggle-line sidebar-toggle-line-last"></span></div></div><aside id="sidebar" class="sidebar"><div class="sidebar-inner"><ul class="sidebar-nav motion-element"><li class="sidebar-nav-toc sidebar-nav-active" data-target="post-toc-wrap">文章目录</li><li class="sidebar-nav-overview" data-target="site-overview-wrap">站点概览</li></ul><section class="site-overview-wrap sidebar-panel"><div class="site-overview"><div class="site-author motion-element" itemprop="author" itemscope itemtype="http://schema.org/Person"><img class="site-author-image" itemprop="image" src="/images/favicon-1536x1536-playpi.png" alt="虾丸派"><p class="site-author-name" itemprop="name">虾丸派</p><p class="site-description motion-element" itemprop="description">记录知识 | 分享技术</p></div><nav class="site-state motion-element"><div class="site-state-item site-state-posts"><a href="/archives/"><span class="site-state-item-count">144</span> <span class="site-state-item-name">日志</span></a></div><div class="site-state-item site-state-categories"><a href="/categories/index.html"><span class="site-state-item-count">13</span> <span class="site-state-item-name">分类</span></a></div><div class="site-state-item site-state-tags"><a href="/tags/index.html"><span class="site-state-item-count">294</span> <span class="site-state-item-name">标签</span></a></div></nav><div class="feed-link motion-element"><a href="/atom.xml" rel="alternate"><i class="fa fa-rss"></i> RSS</a></div><div class="links-of-author motion-element"><span class="links-of-author-item"><a href="https://github.com/iplaypi" target="_blank" title="GitHub"><i class="fa fa-fw fa-github"></i>GitHub</a> </span><span class="links-of-author-item"><a href="https://weibo.com/u/3086148515" target="_blank" title="微博"><i class="fa fa-fw fa-weibo"></i>微博</a> </span><span class="links-of-author-item"><a href="mailto:playpi@qq.com" target="_blank" title="E-Mail"><i class="fa fa-fw fa-envelope"></i>E-Mail</a></span></div><div class="cc-license motion-element" itemprop="license"><a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" class="cc-opacity" target="_blank" rel="external nofollow"><img src="/images/cc-by-nc-sa.svg" alt="Creative Commons"></a></div><div class="links-of-blogroll motion-element links-of-blogroll-inline"><div class="links-of-blogroll-title"><i class="fa fa-fw fa-link"></i> 友情链接</div><ul class="links-of-blogroll-list"><li class="links-of-blogroll-item"><a href="https://github.com/iplaypi" title="GitHub" target="_blank" rel="external nofollow">GitHub</a></li><li class="links-of-blogroll-item"><a href="https://weibo.com/u/3086148515" title="Weibo" target="_blank" rel="external nofollow">Weibo</a></li><li class="links-of-blogroll-item"><a href="https://www.playpi.org" title="虾丸派" target="_blank" rel="external nofollow">虾丸派</a></li><li class="links-of-blogroll-item"><a href="https://www.playpi.org" title="playpi" target="_blank" rel="external nofollow">playpi</a></li><li class="links-of-blogroll-item"><a href="https://www.liaoxuefeng.com" title="廖雪峰" target="_blank" rel="external nofollow">廖雪峰</a></li><li class="links-of-blogroll-item"><a href="http://www.ruanyifeng.com" title="阮一峰" target="_blank" rel="external nofollow">阮一峰</a></li><li class="links-of-blogroll-item"><a href="https://travis-ci.org/iplaypi/iplaypi.github.io" title="travis-ci" target="_blank" rel="external nofollow">travis-ci</a></li><li class="links-of-blogroll-item"><a href="https://www.vultr.com/?ref=7861302-4F" title="Vultr" target="_blank" rel="external nofollow">Vultr</a></li></ul></div></div></section><section class="post-toc-wrap motion-element sidebar-panel sidebar-panel-active"><div class="post-toc"><div class="post-toc-content"><ol class="nav"><li class="nav-item nav-level-1"><a class="nav-link" href="#问题出现"><span class="nav-number">1.</span> <span class="nav-text">问题出现</span></a></li><li class="nav-item nav-level-1"><a class="nav-link" href="#问题解决"><span class="nav-number">2.</span> <span class="nav-text">问题解决</span></a></li><li class="nav-item nav-level-1"><a class="nav-link" href="#问题总结"><span class="nav-number">3.</span> <span class="nav-text">问题总结</span></a></li></ol></div></div></section></div></aside></div></main><footer id="footer" class="footer"><div class="footer-inner"><div class="copyright">© 2016–<span itemprop="copyrightYear">2021</span> <span class="post-meta-divider">|</span> <span class="with-love"><i class="fa fa-heart"></i> </span><span class="author" itemprop="copyrightHolder">虾丸派</span> <span class="post-meta-divider">|</span> <span class="post-meta-item-icon"><i class="fa fa-area-chart"></i> </span><span class="post-meta-item-text">全站字数统计</span> <span title="全站字数统计">326.3k 字</span></div><div class="powered-by">由 <a class="theme-link" target="_blank" href="https://hexo.io" rel="external nofollow">Hexo</a> 强力驱动</div><span class="post-meta-divider">|</span><div class="theme-info">主题 <a class="theme-link" target="_blank" href="https://github.com/iissnan/hexo-theme-next" rel="external nofollow">NexT.Mist</a><script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script><span id="busuanzi_container_site_pv" style="display:none"><span class="post-meta-divider">|</span> 总访问量 <span id="busuanzi_value_site_pv"></span> 次 </span><span id="busuanzi_container_site_uv" style="display:none"><span class="post-meta-divider">|</span> 总访客 <span id="busuanzi_value_site_uv"></span> 人</span></div><div class="busuanzi-count"><script async src="https://dn-lbstatics.qbox.me/busuanzi/2.3/busuanzi.pure.mini.js"></script></div></div></footer><div class="back-to-top"><i class="fa fa-arrow-up"></i> <span id="scrollpercent"><span>0</span>%</span></div></div><script type="text/javascript">"[object Function]"!==Object.prototype.toString.call(window.Promise)&&(window.Promise=null)</script><script type="text/javascript" src="/lib/jquery/index.js?v=2.1.3"></script><script type="text/javascript" src="/lib/fastclick/lib/fastclick.min.js?v=1.0.6"></script><script type="text/javascript" src="/lib/jquery_lazyload/jquery.lazyload.js?v=1.9.7"></script><script type="text/javascript" src="/lib/velocity/velocity.min.js?v=1.2.1"></script><script type="text/javascript" src="/lib/velocity/velocity.ui.min.js?v=1.2.1"></script><script type="text/javascript" src="/lib/fancybox/source/jquery.fancybox.pack.js?v=2.1.5"></script><script type="text/javascript" src="/js/src/utils.js?v=5.1.3"></script><script type="text/javascript" src="/js/src/motion.js?v=5.1.3"></script><script type="text/javascript" src="/js/src/scrollspy.js?v=5.1.3"></script><script type="text/javascript" src="/js/src/post-details.js?v=5.1.3"></script><script type="text/javascript" src="/js/src/bootstrap.js?v=5.1.3"></script><script src="//unpkg.com/valine@1.3.7/dist/Valine.min.js"></script><script type="text/javascript">new Valine({av:AV,el:"#comments",verify:!1,notify:!1,app_id:"FC5Jijeg1meo2K2OzPYWK327-gzGzoHsz",app_key:"6A1ReY8tjhPutK00F01YbJSq",placeholder:"没有问题吗？"})</script><script type="text/javascript">var isfetched=!1,isXml=!0,search_path="search.xml";0===search_path.length?search_path="search.xml":/json$/i.test(search_path)&&(isXml=!1);var path="/"+search_path,onPopupClose=function(t){$(".popup").hide(),$("#local-search-input").val(""),$(".search-result-list").remove(),$("#no-result").remove(),$(".local-search-pop-overlay").remove(),$("body").css("overflow","")};function proceedsearch(){$("body").append('<div class="search-popup-overlay local-search-pop-overlay"></div>').css("overflow","hidden"),$(".search-popup-overlay").click(onPopupClose),$(".popup").toggle();var t=$("#local-search-input");t.attr("autocapitalize","none"),t.attr("autocorrect","off"),t.focus()}var searchFunc=function(t,e,s){"use strict";$("body").append('<div class="search-popup-overlay local-search-pop-overlay"><div id="search-loading-icon"><i class="fa fa-spinner fa-pulse fa-5x fa-fw"></i></div></div>').css("overflow","hidden"),$("#search-loading-icon").css("margin","20% auto 0 auto").css("text-align","center"),$.ajax({url:t,dataType:isXml?"xml":"json",async:!0,success:function(t){isfetched=!0,$(".popup").detach().appendTo(".header-inner");var o=isXml?$("entry",t).map(function(){return{title:$("title",this).text(),content:$("content",this).text(),url:$("url",this).text()}}).get():t,n=document.getElementById(e),r=document.getElementById(s),t=function(){var m=n.value.trim().toLowerCase(),x=m.split(/[\s\-]+/);1<x.length&&x.push(m);var e,w=[];0<m.length&&o.forEach(function(t){var e=!1,o=0,h=0,n=t.title.trim(),r=n.toLowerCase(),s=t.content.trim().replace(/<[^>]+>/g,""),a=s.toLowerCase(),i=decodeURIComponent(t.url),c=[],l=[];if(""!=n&&(x.forEach(function(t){function e(t,e,o){var n=t.length;if(0===n)return[];var r,s=0,a=[];for(o||(e=e.toLowerCase(),t=t.toLowerCase());-1<(r=e.indexOf(t,s));)a.push({position:r,word:t}),s=r+n;return a}c=c.concat(e(t,r,!1)),l=l.concat(e(t,a,!1))}),(0<c.length||0<l.length)&&(e=!0,o=c.length+l.length)),e){function p(t,e,o,n){for(var r=n[n.length-1],s=r.position,a=r.word,i=[],c=0;s+a.length<=o&&0!=n.length;){a===m&&c++,i.push({position:s,length:a.length});var l=s+a.length;for(n.pop();0!=n.length&&(s=(r=n[n.length-1]).position,a=r.word,s<l);)n.pop()}return h+=c,{hits:i,start:e,end:o,searchTextCount:c}}[c,l].forEach(function(t){t.sort(function(t,e){return e.position!==t.position?e.position-t.position:t.word.length-e.word.length})});t=[];0!=c.length&&t.push(p(0,0,n.length,c));for(var u=[];0!=l.length;){var f=l[l.length-1],d=f.position,g=f.word,v=d-20,f=d+80;v<0&&(v=0),(f=f<d+g.length?d+g.length:f)>s.length&&(f=s.length),u.push(p(0,v,f,l))}u.sort(function(t,e){return t.searchTextCount!==e.searchTextCount?e.searchTextCount-t.searchTextCount:t.hits.length!==e.hits.length?e.hits.length-t.hits.length:t.start-e.start});e=parseInt("1");function $(o,t){var n="",r=t.start;return t.hits.forEach(function(t){n+=o.substring(r,t.position);var e=t.position+t.length;n+='<b class="search-keyword">'+o.substring(t.position,e)+"</b>",r=e}),n+=o.substring(r,t.end)}0<=e&&(u=u.slice(0,e));var C="";0!=t.length?C+="<li><a href='"+i+"' class='search-result-title'>"+$(n,t[0])+"</a>":C+="<li><a href='"+i+"' class='search-result-title'>"+n+"</a>",u.forEach(function(t){C+="<a href='"+i+'\'><p class="search-result">'+$(s,t)+"...</p></a>"}),C+="</li>",w.push({item:C,searchTextCount:h,hitCount:o,id:w.length})}}),1===x.length&&""===x[0]?r.innerHTML='<div id="no-result"><i class="fa fa-search fa-5x" /></div>':0===w.length?r.innerHTML='<div id="no-result"><i class="fa fa-frown-o fa-5x" /></div>':(w.sort(function(t,e){return t.searchTextCount!==e.searchTextCount?e.searchTextCount-t.searchTextCount:t.hitCount!==e.hitCount?e.hitCount-t.hitCount:e.id-t.id}),e='<ul class="search-result-list">',w.forEach(function(t){e+=t.item}),e+="</ul>",r.innerHTML=e)};n.addEventListener("input",t),$(".local-search-pop-overlay").remove(),$("body").css("overflow",""),proceedsearch()}})};$(".popup-trigger").click(function(t){t.stopPropagation(),!1===isfetched?searchFunc(path,"local-search-input","local-search-result"):proceedsearch()}),$(".popup-btn-close").click(onPopupClose),$(".popup").click(function(t){t.stopPropagation()}),$(document).on("keyup",function(t){27===t.which&&$(".search-popup").is(":visible")&&onPopupClose()})</script><script>!function(){var t=document.createElement("script"),e=window.location.protocol.split(":")[0];t.src="https"===e?"https://zz.bdstatic.com/linksubmit/push.js":"http://push.zhanzhang.baidu.com/push.js";e=document.getElementsByTagName("script")[0];e.parentNode.insertBefore(t,e)}()</script><script type="text/javascript" src="/js/src/js.cookie.js?v=5.1.3"></script><script type="text/javascript" src="/js/src/scroll-cookie.js?v=5.1.3"></script><script src="/live2dw/lib/L2Dwidget.min.js?094cbace49a39548bed64abff5988b05"></script><script>L2Dwidget.init({pluginRootPath:"live2dw/",pluginJsPath:"lib/",pluginModelPath:"assets/",tagMode:!1,debug:!1,model:{scale:1,jsonPath:"/live2dw/assets/hijiki.model.json"},display:{position:"left",width:100,height:200,hOffset:0,vOffset:-20},mobile:{show:!1,motion:!0,scale:.3},log:!1})</script></body></html>