-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
336 lines (181 loc) · 249 KB
/
atom.xml
File metadata and controls
336 lines (181 loc) · 249 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>抹布先生M</title>
<subtitle>记录技术与生活,keep coding</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="https://blog.monbuilder.top/"/>
<updated>2019-05-06T03:49:08.305Z</updated>
<id>https://blog.monbuilder.top/</id>
<author>
<name>Builder Luo</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>Java常用基础工具库</title>
<link href="https://blog.monbuilder.top/2019/04/28/toolkit01/"/>
<id>https://blog.monbuilder.top/2019/04/28/toolkit01/</id>
<published>2019-04-28T05:38:32.000Z</published>
<updated>2019-05-06T03:49:08.305Z</updated>
<content type="html"><![CDATA[<h1 id="Java常用基础工具库"><a href="#Java常用基础工具库" class="headerlink" title="Java常用基础工具库"></a>Java常用基础工具库</h1><p>下面涉及到的示例代码，皆可以通过Github地址<a href="https://github.com/Builder34/toolkit-demo" target="_blank" rel="noopener">https://github.com/Builder34/toolkit-demo</a> 获取.</p><h2 id="通用工具类-字符串、时间格式化、BeanUtils、IO"><a href="#通用工具类-字符串、时间格式化、BeanUtils、IO" class="headerlink" title="通用工具类(字符串、时间格式化、BeanUtils、IO)"></a>通用工具类(字符串、时间格式化、BeanUtils、IO)</h2><h3 id="1-commons-lang3库"><a href="#1-commons-lang3库" class="headerlink" title="1. commons-lang3库"></a>1. commons-lang3库</h3><h4 id="1-1-org-apache-commons-lang3-StringUtils类"><a href="#1-1-org-apache-commons-lang3-StringUtils类" class="headerlink" title="1.1. org.apache.commons.lang3.StringUtils类"></a>1.1. org.apache.commons.lang3.StringUtils类</h4><p>日常代码中，我们经常和String字符串打交道，经常对字符串进行处理，稍微不注意的话，很容易出现类似NullPointerException这种简单的错误，我们经常写各种if来判断处理这些非业务的逻辑。这时，我们可以利用大牛apache的轮子，通过其StringUtils里面的一些常用方法，改善我们的代码，让我们的业务代码更简洁、优雅。<br>示例代码：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Slf</span>4j</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">AppTest</span> </span>{</span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">stringUtils</span><span class="params">()</span></span>{</span><br><span class="line"> String a = <span class="string">" "</span>;</span><br><span class="line"> String b = <span class="keyword">null</span>;</span><br><span class="line"> <span class="comment">//判断字符对象是否为空以及内容是否为空串(有空格则认为不是空串)</span></span><br><span class="line"> log.info(<span class="string">"StringUtils.isEmpty(a): {}"</span>, StringUtils.isEmpty(a));</span><br><span class="line"> <span class="comment">//判断字符对象是否为空以及内容是否为空串(有空格也会认为是空串)</span></span><br><span class="line"> log.info(<span class="string">"StringUtils.isBlank(a): {}"</span>, StringUtils.isBlank(a));</span><br><span class="line"> <span class="comment">//当b=null时，如果b.trim()则会报空指针异常，使用StringUtils.trim(b)可以避免</span></span><br><span class="line"> log.info(<span class="string">"StringUtils.trim(d): {}"</span>, StringUtils.trim(b));</span><br><span class="line"></span><br><span class="line"> String num = <span class="string">"12.3"</span>;</span><br><span class="line"> <span class="comment">//当b=null时，如果b.trim()则会报空指针异常，使用StringUtils.trim(b)可以避免</span></span><br><span class="line"> log.info(<span class="string">"org.apache.commons.lang3.StringUtils.isNumericSpace(): {} isNumber: {}"</span>, num, StringUtils.isNumericSpace(b));</span><br><span class="line"> log.info(<span class="string">"com.alibaba.druid.util.StringUtils.isNumber(): {} isNumber: {}"</span>, num, com.alibaba.druid.util.StringUtils.isNumber(num));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><em>注意：common-lang3中的StringUtils. isNumeric()或isNumericSpace()并不能判断字符串中带小数点的数字值字符为数字。可以通过com.alibaba.druid.util.StringUtils.isNumber(str)，此方法来判断。</em></p><h4 id="1-2-org-apache-commons-lang3-time-DateFormatUtils-DateUtils"><a href="#1-2-org-apache-commons-lang3-time-DateFormatUtils-DateUtils" class="headerlink" title="1.2 org.apache.commons.lang3.time.DateFormatUtils/DateUtils"></a>1.2 org.apache.commons.lang3.time.DateFormatUtils/DateUtils</h4><p>时间转换工具类:</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">dateFormatUtils</span><span class="params">()</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> String pattern = <span class="string">"yyyy-MM-dd HH:mm:ss"</span>;</span><br><span class="line"> String timeStr = DateFormatUtils.format(<span class="keyword">new</span> Date(), pattern);</span><br><span class="line"> <span class="keyword">long</span> timestamp = DateUtils.parseDate(timeStr, pattern).getTime();</span><br><span class="line"> log.info(<span class="string">"==> current time: {}"</span>, timeStr);</span><br><span class="line"> log.info(<span class="string">"==> current time timestamp: {}"</span>, timestamp);</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><h4 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h4><ul><li>StringUtils.isEmpty(str)/StringUtils.isNotEmpty(str): 判断字符对象是否为null或空串(有空格则认为不是空串)</li><li>StringUtils.isBlank(str)/StringUtils.isNotBlank(str): 判断字符对象是否为null或空串(有空格也会认为是空串)</li><li>DateFormatUtils.format(date, pattern): 将Date时间对象按表达式的格式转换成时间字符串</li><li>DateUtils.parseDate(timeStr, pattern): 将时间字符串反转成Date对象</li><li>ToStringBuilder.reflectionToString(obj): 将对象内容转换成字符串输出(下一节有使用到)</li><li>…</li></ul><p>对于学习某个工具类，我们可以通过Intellij IDEA中可通过打开此类的源代码，然后通过快捷键(MacOS: command+7; Windows: Alt+7)打开查看类方法列表(Structure)，从方法名字上大概可以看出具体有那些适合自己使用的方法。<br><img src="/2019/04/28/toolkit01/2.png" alt="StringUtils"></p><p>以上示例使用到的jar包可通过maven的pom.xml文件依赖导入:</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.commons<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>commons-lang3<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>3.8.1<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.alibaba<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>druid<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.1.9<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br></pre></td></tr></table></figure><h3 id="2-common-beanutils库"><a href="#2-common-beanutils库" class="headerlink" title="2.common-beanutils库"></a>2.common-beanutils库</h3><p>Map、JavaBean是我们日常业务代码中经常使用到的2种类，有时因为业务原因，Map、JavaBean需要相互转换copy啥的操作时，如果手动set/put，字段多的时候，就要吐血了。这里我们推荐使用BeanUtils来简化我们的代码</p><h4 id="2-1-org-apache-commons-beanutils-BeanUtils类"><a href="#2-1-org-apache-commons-beanutils-BeanUtils类" class="headerlink" title="2.1.org.apache.commons.beanutils.BeanUtils类"></a>2.1.org.apache.commons.beanutils.BeanUtils类</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">beanUtils</span><span class="params">()</span> <span class="keyword">throws</span> InvocationTargetException, IllegalAccessException </span>{</span><br><span class="line"> CompanyBean bean = <span class="keyword">new</span> CompanyBean();</span><br><span class="line"> bean.setId(<span class="number">1</span>);</span><br><span class="line"> bean.setName(<span class="string">"中国移动广州分公司"</span>);</span><br><span class="line"> bean.setAddress(<span class="string">"广州市天河区中山大道"</span>);</span><br><span class="line"> bean.setTel(<span class="string">"020-10086000"</span>);</span><br><span class="line"> CompanyBean destObj = <span class="keyword">new</span> CompanyBean();</span><br><span class="line"> <span class="comment">//复制bean之间复制内容, 新对象destObj需要先实例化</span></span><br><span class="line"> BeanUtils.copyProperties(destObj, bean);</span><br><span class="line"> <span class="comment">//ToStringBuilder类来自commons-lang3库：将对象内容转换成字符串输出，方便于日志输出</span></span><br><span class="line"> log.info(<span class="string">"destObj from BeanUtils.copyProperties: {}"</span>, ToStringBuilder.reflectionToString(destObj));</span><br><span class="line"> </span><br><span class="line"> Map<String, Object> map = <span class="keyword">new</span> HashMap<>();</span><br><span class="line"> map.put(<span class="string">"id"</span>, <span class="number">2</span>);</span><br><span class="line"> map.put(<span class="string">"name"</span>, <span class="string">"中国联通广州分公司"</span>);</span><br><span class="line"> map.put(<span class="string">"address"</span>, <span class="string">"广州市天河区中山大道2号"</span>);</span><br><span class="line"> map.put(<span class="string">"tel"</span>, <span class="string">"020-10000110"</span>);</span><br><span class="line"> <span class="comment">//将map(key,value)映射成bean</span></span><br><span class="line"> BeanUtils.populate(destObj, map);</span><br><span class="line"> log.info(<span class="string">"destObj from BeanUtils.populate: {}"</span>, ToStringBuilder.reflectionToString(destObj));</span><br><span class="line"> <span class="comment">//复制对象，与copyProperties()方法比较，这里新对象可以不先实例化</span></span><br><span class="line"> CompanyBean cloneBean = (CompanyBean)BeanUtils.cloneBean(destObj);</span><br><span class="line"> log.info(<span class="string">"cloneBean from BeanUtils.cloneBean: {}"</span>, ToStringBuilder.reflectionToString(cloneBean));</span><br><span class="line"> <span class="comment">//将JavaBean转换成Map</span></span><br><span class="line"> Map newMap = BeanUtils.describe(cloneBean);</span><br><span class="line"> log.info(<span class="string">"newMap from BeanUtils.describe: {}"</span>, <span class="keyword">new</span> Gson().toJson(newMap));</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>日志输出如下：</p><figure class="highlight verilog"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">19</span> <span class="number">13</span>:<span class="number">17</span>:<span class="number">21</span><span class="variable">.064</span> [main] INFO com<span class="variable">.monbuilder</span><span class="variable">.AppTest</span> - destObj from BeanUtils<span class="variable">.copyProperties</span>: com<span class="variable">.monbuilder</span><span class="variable">.bean</span><span class="variable">.CompanyBean</span>@<span class="number">10683</span>d9d[id=<span class="number">1</span>,name=中国移动广州分公司,address=广州市天河区中山大道<span class="number">1</span>号,tel=<span class="number">020</span>-<span class="number">10086000</span>]</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">19</span> <span class="number">13</span>:<span class="number">17</span>:<span class="number">21</span><span class="variable">.070</span> [main] INFO com<span class="variable">.monbuilder</span><span class="variable">.AppTest</span> - destObj from BeanUtils<span class="variable">.populate</span>: com<span class="variable">.monbuilder</span><span class="variable">.bean</span><span class="variable">.CompanyBean</span>@<span class="number">10683</span>d9d[id=<span class="number">2</span>,name=中国联通广州分公司,address=广州市天河区中山大道<span class="number">2</span>号,tel=<span class="number">020</span>-<span class="number">10000110</span>]</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">19</span> <span class="number">13</span>:<span class="number">48</span>:<span class="number">14</span><span class="variable">.966</span> [main] INFO com<span class="variable">.monbuilder</span><span class="variable">.AppTest</span> - newMap from BeanUtils<span class="variable">.describe</span>: {<span class="string">"address"</span>:<span class="string">"广州市天河区中山大道2号"</span>,<span class="string">"name"</span>:<span class="string">"中国联通广州分公司"</span>,<span class="string">"tel"</span>:<span class="string">"020-10000110"</span>,<span class="string">"id"</span>:<span class="string">"2"</span>,<span class="string">"class"</span>:<span class="string">"class com.monbuilder.bean.CompanyBean"</span>}</span><br></pre></td></tr></table></figure><h4 id="总结-1"><a href="#总结-1" class="headerlink" title="总结"></a>总结</h4><ul><li>BeanUtils.copyProperties(destObj, sourceObj): JavaBean之间内容的复制</li><li>BeanUtils.cloneBean(obj): 复制对象</li><li>BeanUtils.populate(destObj, sourceMap): Map转换成JavaBean</li><li>BeanUtils.describe(bean): 将JavaBean转换成Map</li></ul><h3 id="3-commons-io库"><a href="#3-commons-io库" class="headerlink" title="3.commons-io库"></a>3.commons-io库</h3><h4 id="org-apache-commons-io-IOUtils类"><a href="#org-apache-commons-io-IOUtils类" class="headerlink" title="org.apache.commons.io.IOUtils类"></a>org.apache.commons.io.IOUtils类</h4><p>这个io工具类非常有用，当我们在处理流的过程中，经常需要把流与字节数组之间相互转换，以及在处理完之后，关闭流等等这些操作时，我们需要写挺多处理逻辑，close时还需要写if判空啥的，但是使用了这个IOUtil后，我们的处理代码或简洁非常多的。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">ioUtils</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> InputStream io = <span class="keyword">this</span>.getClass().getClassLoader().getResourceAsStream(<span class="string">"README.md"</span>);</span><br><span class="line"> BufferedReader br = <span class="keyword">new</span> BufferedReader(<span class="keyword">new</span> InputStreamReader(io));</span><br><span class="line"> log.info(<span class="string">"==> IOUtils.toString(br): {}"</span>, IOUtils.toString(br));</span><br><span class="line"> IOUtils.closeQuietly(br);</span><br><span class="line"> IOUtils.closeQuietly(io);</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><figure class="highlight verilog"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">19</span> <span class="number">14</span>:<span class="number">29</span>:<span class="number">26</span><span class="variable">.140</span> [main] INFO com<span class="variable">.monbuilder</span><span class="variable">.AppTest</span> - ==> IOUtils<span class="variable">.toString</span>(br): toolkit-demo,工具类库使用示例</span><br></pre></td></tr></table></figure><p><em>上面只是简单的展示将文件流内容转换成字符串，之后再关闭流，是不是非常简洁呢？IOUtils里面还有非常多的好方法可以使用，这些可以根据自己在具体的工作场景下，查看IOUtils的方法列表，找到自己需要的方法</em></p><p><img src="/2019/04/28/toolkit01/1.png" alt="IOUtils"></p><h4 id="总结-2"><a href="#总结-2" class="headerlink" title="总结"></a>总结</h4><p>IOUtils常用的方法有：</p><ul><li>IOUtils.closeQuietly(obj): 可关闭流\Socket\SocketServer等多种对象</li><li>IOUtils.copy(InputStream, Writer): 复制输入流</li><li>IOUtils.write(byte[], OutputStream): 将字节数组转换成流</li><li>IOUtils.toByteArray(InputStream): 将输入流转换成字节数组</li><li>IOUtils.toInputStream(String): 将字符串转换成输入流</li><li>IOUtils.toString(InputStream): 将输入流转换成字符串</li></ul><p>上面介绍的都是来自于apache官方的类库，还有好多类与方法有待我们发掘使用。另外，还介绍另1个非常有名的工具类库：guava，来自于Google；功能也有类似的，当然也有很多的扩展使用，本文就不再详细介绍了。</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.google.guava<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>guava<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>23.2-jre<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br></pre></td></tr></table></figure><h2 id="JSON工具类"><a href="#JSON工具类" class="headerlink" title="JSON工具类"></a>JSON工具类</h2><p>在如今REST API盛行的年代，前后端分离成为标配，json成为两者之间的传输桥梁，在我们工作做打交道肯定非常频繁咯。在Java开发中，JSON工具类的选择并不少：</p><ul><li>fastjson 阿里巴巴出品，转换快，但表现不够稳定</li><li>Gson google出品，使用简单，轻量</li><li>Jackson spring官方使用，性能快，转换快，配置更灵活，不同场景下表现更稳定</li></ul><p>比较推荐使用的是Jackson，如果我们使用Spring Boot或Spring Cloud时，构建web项目里面内置的json库就是Jackson库。</p><p>下面通过Jackson库简单封装成JavaBean/json互转的工具类：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> com.monbuilder.util;</span><br><span class="line"><span class="keyword">import</span> com.fasterxml.jackson.annotation.JsonInclude;</span><br><span class="line"><span class="keyword">import</span> com.fasterxml.jackson.core.JsonParser;</span><br><span class="line"><span class="keyword">import</span> com.fasterxml.jackson.databind.DeserializationFeature;</span><br><span class="line"><span class="keyword">import</span> com.fasterxml.jackson.databind.ObjectMapper;</span><br><span class="line"><span class="keyword">import</span> lombok.extern.slf4j.Slf4j;</span><br><span class="line"><span class="keyword">import</span> java.text.SimpleDateFormat;</span><br><span class="line"><span class="comment">/***</span></span><br><span class="line"><span class="comment"> * JSON转换工具类</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> <a href="mailto:lcbiao34@gmail.com">Builder34</a></span></span><br><span class="line"><span class="comment"> * <span class="doctag">@date</span> 2018-11-01 11:14:26</span></span><br><span class="line"><span class="comment"> * */</span></span><br><span class="line"><span class="meta">@Slf</span>4j</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">JacksonUtil</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> ObjectMapper objectMapper = <span class="keyword">new</span> ObjectMapper();</span><br><span class="line"> <span class="keyword">static</span> {</span><br><span class="line"> objectMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, <span class="keyword">true</span>);</span><br><span class="line"> objectMapper.configure(JsonParser.Feature.ALLOW_SINGLE_QUOTES, <span class="keyword">true</span>);</span><br><span class="line"> objectMapper.configure(JsonParser.Feature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER, <span class="keyword">true</span>);</span><br><span class="line"> objectMapper.setDateFormat(<span class="keyword">new</span> SimpleDateFormat(<span class="string">"yyyy-MM-dd HH:mm:ss"</span>));</span><br><span class="line"> <span class="comment">// 设置输入时忽略JSON字符串中存在而Java对象实际没有的属性</span></span><br><span class="line"> objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, <span class="keyword">false</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">//设置不输出值为 null 的属性</span></span><br><span class="line"> objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 将JSON字符串根据指定的Class反序列化成Java对象(转换过程中出现异常则返回null对象)</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> json JSON字符串</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> objClass JavaObject对象Class</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> 反序列化生成的Java对象</span></span><br><span class="line"><span class="comment"> * */</span></span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <T> <span class="function">T <span class="title">toJavaObject</span><span class="params">(String json, Class<T> objClass)</span> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">return</span> objectMapper.readValue(json, objClass);</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> log.error(<span class="string">""</span>, e);</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">null</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 将Java对象序列化成JSON字符串(转换过程中出现异常则返回空对象json串"{}")</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> obj 待序列化生成JSON字符串的Java对象</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> JSON字符串</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">toJsonString</span><span class="params">(Object obj)</span> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">return</span> objectMapper.writeValueAsString(obj);</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> log.error(<span class="string">""</span>,e);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"{}"</span>;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>运用上面的工具类，示例：</p><figure class="highlight mipsasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">@Test</span><br><span class="line"> public void <span class="keyword">jsonUtils(){</span></span><br><span class="line"><span class="keyword"> </span> CompanyBean <span class="keyword">bean </span>= new CompanyBean()<span class="comment">;</span></span><br><span class="line"> <span class="keyword">bean.setId(1);</span></span><br><span class="line"><span class="keyword"> </span> <span class="keyword">bean.setName("中国移动广州分公司");</span></span><br><span class="line"><span class="keyword"> </span> <span class="keyword">bean.setAddress("广州市天河区中山大道1号");</span></span><br><span class="line"><span class="keyword"> </span> <span class="keyword">bean.setTel("020-10086000");</span></span><br><span class="line"><span class="keyword"> </span> log.info(<span class="string">"==> JacksonUtil.toJsonString(bean): {}"</span>, <span class="keyword">JacksonUtil.toJsonString(bean));</span></span><br><span class="line"><span class="keyword"> </span> String <span class="keyword">json </span>= <span class="keyword">JacksonUtil.toJsonString(bean);</span></span><br><span class="line"><span class="keyword"> </span> log.info(<span class="string">"==> JacksonUtil.toJavaObject: {}"</span>, ToStringBuilder.reflectionToString(<span class="keyword">JacksonUtil.toJavaObject(json, </span>CompanyBean.class)))<span class="comment">;</span></span><br><span class="line"></span><br><span class="line"> }</span><br></pre></td></tr></table></figure><figure class="highlight verilog"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">19</span> <span class="number">15</span>:<span class="number">42</span>:<span class="number">16</span><span class="variable">.081</span> [main] INFO com<span class="variable">.monbuilder</span><span class="variable">.AppTest</span> - ==> JacksonUtil<span class="variable">.toJsonString</span>(bean): {<span class="string">"id"</span>:<span class="number">1</span>,<span class="string">"name"</span>:<span class="string">"中国移动广州分公司"</span>,<span class="string">"address"</span>:<span class="string">"广州市天河区中山大道1号"</span>,<span class="string">"tel"</span>:<span class="string">"020-10086000"</span>}</span><br><span class="line"><span class="number">2019</span>-<span class="number">01</span>-<span class="number">19</span> <span class="number">15</span>:<span class="number">42</span>:<span class="number">16</span><span class="variable">.144</span> [main] INFO com<span class="variable">.monbuilder</span><span class="variable">.AppTest</span> - ==> JacksonUtil<span class="variable">.toJavaObject</span>: com<span class="variable">.monbuilder</span><span class="variable">.bean</span><span class="variable">.CompanyBean</span>@<span class="number">376</span>a0d86[id=<span class="number">1</span>,name=中国移动广州分公司,address=广州市天河区中山大道<span class="number">1</span>号,tel=<span class="number">020</span>-<span class="number">10086000</span>]</span><br></pre></td></tr></table></figure><p><strong>最后想要说的是，轮子非常多，当然我们可以重复造轮子，但是我们更提倡的是使用已有优秀的轮子(工具类)，开发出更简洁、优雅、业务逻辑更清晰的代码。</strong></p>]]></content>
<summary type="html">
<h1 id="Java常用基础工具库"><a href="#Java常用基础工具库" class="headerlink" title="Java常用基础工具库"></a>Java常用基础工具库</h1><p>下面涉及到的示例代码，皆可以通过Github地址<a href="h
</summary>
<category term="commons utils" scheme="https://blog.monbuilder.top/categories/commons-utils/"/>
<category term="Java工具库" scheme="https://blog.monbuilder.top/tags/Java%E5%B7%A5%E5%85%B7%E5%BA%93/"/>
</entry>
<entry>
<title>Elasticsearch学习教程系列(2)-命令学习(二)批处理、数据操作、搜索</title>
<link href="https://blog.monbuilder.top/2019/03/14/elasticsearch-learn02/"/>
<id>https://blog.monbuilder.top/2019/03/14/elasticsearch-learn02/</id>
<published>2019-03-14T12:08:49.000Z</published>
<updated>2019-03-14T11:10:37.846Z</updated>
<content type="html"><![CDATA[<p># </p><p>上一篇文章中，我们介绍了Elasticsearch集群运行情况、索引、文档的CRUD操作了，下面让我们来愉快地学习一些新的命令吧。</p><h2 id="批处理"><a href="#批处理" class="headerlink" title="批处理"></a>批处理</h2><p>Elasticsearch除了能够索引，更新和删除单个文档之外，还提供了使用_bulkAPI批量执行上述任何操作的功能。<em>此功能非常重要，因为它提供了一种非常有效的机制，可以尽可能快地执行多个操作，并尽可能少地进行网络往返。</em><br>作为一个简单示例，以下调用在一个批量操作中索引两个文档（ID 1 - John Doe和ID 2 - Jane Doe）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br></pre></td><td class="code"><pre><span class="line">[builder @master~] $ curl - X POST <span class="string">"localhost:9200/customer/_doc/_bulk?pretty"</span> - H <span class="string">'Content-Type: application/json'</span> - d <span class="string">'</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "index":</span></span><br><span class="line"><span class="string"> {</span></span><br><span class="line"><span class="string"> "_id": "1"</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "name": "John Doe"</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "index":</span></span><br><span class="line"><span class="string"> {</span></span><br><span class="line"><span class="string"> "_id": "2"</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "name": "Jane Doe"</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">'</span></span><br><span class="line">{</span><br><span class="line"> <span class="string">"took"</span>: 19,</span><br><span class="line"> <span class="string">"errors"</span>: <span class="literal">false</span>,</span><br><span class="line"> <span class="string">"items"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"index"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_index"</span>: <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span>: <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span>: <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"_version"</span>: 6,</span><br><span class="line"> <span class="string">"result"</span>: <span class="string">"updated"</span>,</span><br><span class="line"> <span class="string">"_shards"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"total"</span>: 2,</span><br><span class="line"> <span class="string">"successful"</span>: 1,</span><br><span class="line"> <span class="string">"failed"</span>: 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span>: 5,</span><br><span class="line"> <span class="string">"_primary_term"</span>: 2,</span><br><span class="line"> <span class="string">"status"</span>: 200</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"index"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_index"</span>: <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span>: <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span>: <span class="string">"2"</span>,</span><br><span class="line"> <span class="string">"_version"</span>: 1,</span><br><span class="line"> <span class="string">"result"</span>: <span class="string">"created"</span>,</span><br><span class="line"> <span class="string">"_shards"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"total"</span>: 2,</span><br><span class="line"> <span class="string">"successful"</span>: 1,</span><br><span class="line"> <span class="string">"failed"</span>: 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span>: 0,</span><br><span class="line"> <span class="string">"_primary_term"</span>: 2,</span><br><span class="line"> <span class="string">"status"</span>: 201</span><br><span class="line"> }</span><br><span class="line"> }]</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>下面示例更新第一个文档（ ID为1）， 然后在一个批量操作中删除第二个文档（ ID为2）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br></pre></td><td class="code"><pre><span class="line">[builder @master~] $ curl - X POST <span class="string">"localhost:9200/customer/_doc/_bulk?pretty"</span> - H <span class="string">'Content-Type: application/json'</span> - d <span class="string">'</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "index":</span></span><br><span class="line"><span class="string"> {</span></span><br><span class="line"><span class="string"> "_id": "1"</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "name": "John Doe"</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "index":</span></span><br><span class="line"><span class="string"> {</span></span><br><span class="line"><span class="string"> "_id": "2"</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "name": "Jane Doe"</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"><span class="string">'</span></span><br><span class="line">{</span><br><span class="line"> <span class="string">"took"</span>: 15,</span><br><span class="line"> <span class="string">"errors"</span>: <span class="literal">false</span>,</span><br><span class="line"> <span class="string">"items"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"index"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_index"</span>: <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span>: <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span>: <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"_version"</span>: 7,</span><br><span class="line"> <span class="string">"result"</span>: <span class="string">"updated"</span>,</span><br><span class="line"> <span class="string">"_shards"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"total"</span>: 2,</span><br><span class="line"> <span class="string">"successful"</span>: 1,</span><br><span class="line"> <span class="string">"failed"</span>: 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span>: 6,</span><br><span class="line"> <span class="string">"_primary_term"</span>: 2,</span><br><span class="line"> <span class="string">"status"</span>: 200</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"index"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_index"</span>: <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span>: <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span>: <span class="string">"2"</span>,</span><br><span class="line"> <span class="string">"_version"</span>: 2,</span><br><span class="line"> <span class="string">"result"</span>: <span class="string">"updated"</span>,</span><br><span class="line"> <span class="string">"_shards"</span>:</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"total"</span>: 2,</span><br><span class="line"> <span class="string">"successful"</span>: 1,</span><br><span class="line"> <span class="string">"failed"</span>: 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span>: 1,</span><br><span class="line"> <span class="string">"_primary_term"</span>: 2,</span><br><span class="line"> <span class="string">"status"</span>: 200</span><br><span class="line"> }</span><br><span class="line"> }]</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>请注意，对于删除操作，之后没有相应的源文档，因为删除只需要删除文档的ID。<br>Bulk API不会因其中一个操作失败而失败。如果单个操作因任何原因失败，它将继续处理其后的其余操作。批量API返回时，它将为每个操作提供一个状态（按照发送的顺序），以便您可以检查特定操作是否失败。</p><h2 id="数据操作"><a href="#数据操作" class="headerlink" title="数据操作"></a>数据操作</h2><h3 id="导入数据"><a href="#导入数据" class="headerlink" title="导入数据"></a>导入数据</h3><p>下面我们在某个文件夹中保存着1个json文件，内容如下：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~/Developer/esTempData]$ cat accounts.json</span><br><span class="line">{<span class="string">"index"</span>:{<span class="string">"_id"</span>:<span class="string">"1"</span>}}</span><br><span class="line">{<span class="string">"account_number"</span>:1,<span class="string">"balance"</span>:39225,<span class="string">"firstname"</span>:<span class="string">"Amber"</span>,<span class="string">"lastname"</span>:<span class="string">"Duke"</span>,<span class="string">"age"</span>:32,<span class="string">"gender"</span>:<span class="string">"M"</span>,<span class="string">"address"</span>:<span class="string">"880 Holmes Lane"</span>,<span class="string">"employer"</span>:<span class="string">"Pyrami"</span>,<span class="string">"email"</span>:<span class="string">"amberduke@pyrami.com"</span>,<span class="string">"city"</span>:<span class="string">"Brogan"</span>,<span class="string">"state"</span>:<span class="string">"IL"</span>}</span><br><span class="line"></span><br><span class="line">{<span class="string">"index"</span>:{<span class="string">"_id"</span>:<span class="string">"6"</span>}}</span><br><span class="line">{<span class="string">"account_number"</span>:6,<span class="string">"balance"</span>:5686,<span class="string">"firstname"</span>:<span class="string">"Hattie"</span>,<span class="string">"lastname"</span>:<span class="string">"Bond"</span>,<span class="string">"age"</span>:36,<span class="string">"gender"</span>:<span class="string">"M"</span>,<span class="string">"address"</span>:<span class="string">"671 Bristol Street"</span>,<span class="string">"employer"</span>:<span class="string">"Netagy"</span>,<span class="string">"email"</span>:<span class="string">"hattiebond@netagy.com"</span>,<span class="string">"city"</span>:<span class="string">"Dante"</span>,<span class="string">"state"</span>:<span class="string">"TN"</span>}</span><br><span class="line">...</span><br><span class="line"><span class="comment"># 我们的json文件中有1000条数据，上面只展示2条数据</span></span><br></pre></td></tr></table></figure><p>导入我们的json文件数据到Elasticsearch，需要在json文件的当前路径下执行：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~/Developer/esTempData]$ curl -H <span class="string">"Content-Type: application/json"</span> -XPOST <span class="string">"localhost:9200/bank/_doc/_bulk?pretty&refresh"</span> --data-binary <span class="string">"@accounts.json"</span></span><br><span class="line">...</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">## 查看索引数据，可以看到第2行的数据有1000条记录了</span></span><br><span class="line">[builder@master ~]$ curl -X GET <span class="string">"http://localhost:9200/_cat/indices?v"</span></span><br><span class="line">health status index uuid pri rep docs.count docs.deleted store.size pri.store.size</span><br><span class="line">green open .kibana_1 WShElb71RVigvhHRsU5vIA 1 0 3 0 11.9kb 11.9kb</span><br><span class="line">yellow open bank LLrKZKoNT-ifFpiv-dBc9w 5 1 1000 0 474.6kb 474.6kb</span><br><span class="line">yellow open customer jrkOIUCjTLec7_5OcwldYw 5 1 3 0 10.9kb 10.9kb</span><br></pre></td></tr></table></figure><h2 id="搜索API"><a href="#搜索API" class="headerlink" title="搜索API"></a>搜索API</h2><p>搜索有两种基本方式：一种是通过发送搜索参数<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.5/search-uri-request.html" target="_blank" rel="noopener">REST请求URI</a>和其他通过发送他们<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.5/search-request-body.html" target="_blank" rel="noopener">REST请求JSON主体</a>。请求JSON体方法更具表现力，并以更可读的JSON格式定义搜索。我们将尝试一个请求URI方法的示例，但是对于本教程的其余部分，我们将专门使用请求体方法。</p><p>可以从<code>_search</code>端点访问用于搜索的REST API 。此示例返回<code>bank</code>索引中的所有文档：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X GET <span class="string">'localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty'</span></span><br></pre></td></tr></table></figure><p>该<code>q=*</code>参数指示Elasticsearch匹配索引中的所有文档。该<code>sort=account_number:asc</code>参数指示使用<code>account_number</code>每个文档的字段以升序对结果进行排序。该<code>pretty</code>参数再次告诉Elasticsearch返回漂亮的JSON结果。</p><p>响应（部分显示）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="string">"took"</span> : 63,</span><br><span class="line"> <span class="string">"timed_out"</span> : <span class="literal">false</span>,</span><br><span class="line"> <span class="string">"_shards"</span> : {</span><br><span class="line"> <span class="string">"total"</span> : 5,</span><br><span class="line"> <span class="string">"successful"</span> : 5,</span><br><span class="line"> <span class="string">"skipped"</span> : 0,</span><br><span class="line"> <span class="string">"failed"</span> : 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"hits"</span> : {</span><br><span class="line"> <span class="string">"total"</span> : 1000,</span><br><span class="line"> <span class="string">"max_score"</span> : null,</span><br><span class="line"> <span class="string">"hits"</span> : [ {</span><br><span class="line"> <span class="string">"_index"</span> : <span class="string">"bank"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"0"</span>,</span><br><span class="line"> <span class="string">"sort"</span>: [0],</span><br><span class="line"> <span class="string">"_score"</span> : null,</span><br><span class="line"> <span class="string">"_source"</span> : {<span class="string">"account_number"</span>:0,<span class="string">"balance"</span>:16623,<span class="string">"firstname"</span>:<span class="string">"Bradshaw"</span>,<span class="string">"lastname"</span>:<span class="string">"Mckenzie"</span>,<span class="string">"age"</span>:29,<span class="string">"gender"</span>:<span class="string">"F"</span>,<span class="string">"address"</span>:<span class="string">"244 Columbus Place"</span>,<span class="string">"employer"</span>:<span class="string">"Euron"</span>,<span class="string">"email"</span>:<span class="string">"bradshawmckenzie@euron.com"</span>,<span class="string">"city"</span>:<span class="string">"Hobucken"</span>,<span class="string">"state"</span>:<span class="string">"CO"</span>}</span><br><span class="line"> }, {</span><br><span class="line"> <span class="string">"_index"</span> : <span class="string">"bank"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"sort"</span>: [1],</span><br><span class="line"> <span class="string">"_score"</span> : null,</span><br><span class="line"> <span class="string">"_source"</span> : {<span class="string">"account_number"</span>:1,<span class="string">"balance"</span>:39225,<span class="string">"firstname"</span>:<span class="string">"Amber"</span>,<span class="string">"lastname"</span>:<span class="string">"Duke"</span>,<span class="string">"age"</span>:32,<span class="string">"gender"</span>:<span class="string">"M"</span>,<span class="string">"address"</span>:<span class="string">"880 Holmes Lane"</span>,<span class="string">"employer"</span>:<span class="string">"Pyrami"</span>,<span class="string">"email"</span>:<span class="string">"amberduke@pyrami.com"</span>,<span class="string">"city"</span>:<span class="string">"Brogan"</span>,<span class="string">"state"</span>:<span class="string">"IL"</span>}</span><br><span class="line"> }, ...</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>关于response响应字段含义：</p><ul><li><code>took</code> - Elasticsearch执行搜索的时间（以毫秒为单位）</li><li><code>timed_out</code> - 告诉我们搜索是否超时</li><li><code>_shards</code> - 告诉我们搜索了多少个分片，以及搜索成功/失败分片的计数</li><li><code>hits</code> - 搜索结果</li><li><code>hits.total</code> - 符合我们搜索条件的文档总数</li><li><code>hits.hits</code> - 实际的搜索结果数组（默认为前10个文档）</li><li><code>hits.sort</code> - 对结果进行排序键（如果按分数排序则丢失）</li><li><code>hits._score</code>并<code>max_score</code>- 暂时忽略这些字段</li></ul><p>下面使用JSON请求体的方法完成上述相同的搜索：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ curl -X GET <span class="string">'localhost:9200/bank/_search'</span> -d <span class="string">'{</span></span><br><span class="line"><span class="string"> "query": { "match_all": {} },</span></span><br><span class="line"><span class="string"> "sort": [</span></span><br><span class="line"><span class="string"> { "account_number": "asc" }</span></span><br><span class="line"><span class="string"> ]</span></span><br><span class="line"><span class="string">}'</span> -H <span class="string">"Content-Type:application/json"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># response 报文:</span></span><br><span class="line">{<span class="string">"took"</span>:14,<span class="string">"timed_out"</span>:<span class="literal">false</span>,<span class="string">"_shards"</span>:{<span class="string">"total"</span>:5,<span class="string">"successful"</span>:5,<span class="string">"skipped"</span>:0,<span class="string">"failed"</span>:0},<span class="string">"hits"</span>:{<span class="string">"total"</span>:1000,<span class="string">"max_score"</span>:null,<span class="string">"hits"</span>:[{<span class="string">"_index"</span>:<span class="string">"bank"</span>,<span class="string">"_type"</span>:<span class="string">"_doc"</span>,<span class="string">"_id"</span>:<span class="string">"0"</span>,<span class="string">"_score"</span>:null,<span class="string">"_source"</span>:{<span class="string">"account_number"</span>:0,<span class="string">"balance"</span>:16623,<span class="string">"firstname"</span>:<span class="string">"Bradshaw"</span>,<span class="string">"lastname"</span>:<span class="string">"Mckenzie"</span>,<span class="string">"age"</span>:29,<span class="string">"gender"</span>:<span class="string">"F"</span>,<span class="string">"address"</span>:<span class="string">"244 Columbus Place"</span>,<span class="string">"employer"</span>:<span class="string">"Euron"</span>,<span class="string">"email"</span>:<span class="string">"bradshawmckenzie@euron.com"</span>,<span class="string">"city"</span>:<span class="string">"Hobucken"</span>,<span class="string">"state"</span>:<span class="string">"CO"</span>},<span class="string">"sort"</span>:[0]}...</span><br></pre></td></tr></table></figure><p>好啦，本文暂时介绍这几个命令的简单使用方法。更多elasticsearch的相关知识我们在后面再娓娓道来…</p>]]></content>
<summary type="html">
<p># </p>
<p>上一篇文章中，我们介绍了Elasticsearch集群运行情况、索引、文档的CRUD操作了，下面让我们来愉快地学习一些新的命令吧。</p>
<h2 id="批处理"><a href="#批处理" class="headerlink" title="批处理
</summary>
<category term="搜索引擎" scheme="https://blog.monbuilder.top/categories/%E6%90%9C%E7%B4%A2%E5%BC%95%E6%93%8E/"/>
<category term="Elasticsearch" scheme="https://blog.monbuilder.top/tags/Elasticsearch/"/>
</entry>
<entry>
<title>Elasticsearch学习教程系列(1)-命令学习(一) 集群健康、索引、文档操作</title>
<link href="https://blog.monbuilder.top/2019/03/14/elasticsearch-learn01/"/>
<id>https://blog.monbuilder.top/2019/03/14/elasticsearch-learn01/</id>
<published>2019-03-14T04:15:22.000Z</published>
<updated>2019-03-14T03:07:38.915Z</updated>
<content type="html"><![CDATA[<p><em>本教程是基于Elasticsearch6.5版本编写</em><br>在本系列教程上一篇文章中，我们介绍了Elasticsearch的一些基本概念、安装并运行起了一个Elasticsearch节点。现在我们已经启动并运行了节点（集群），下一步是了解如何与它进行通信。幸运的是，Elasticsearch提供了一个非常全面和强大的REST API，您可以使用它与集群进行交互。使用API 可以完成的一些事项如下：</p><ul><li>检查集群运行情况，状态和统计信息</li><li>管理您的群集，节点和索引数据和元数据</li><li>对索引执行CRUD（创建，读取，更新和删除）和搜索操作</li><li>执行高级搜索操作，例如分页，排序，过滤，脚本编写，聚合等等</li></ul><h2 id="集群运行情况-Cluster-Health"><a href="#集群运行情况-Cluster-Health" class="headerlink" title="集群运行情况(Cluster Health)"></a>集群运行情况(Cluster Health)</h2><p>让我们从基本运行状况检查开始，我们可以使用它来查看集群的运行情况。我们将使用curl来执行此操作，但您可以使用任何允许您进行HTTP / REST调用的工具。假设我们仍然在我们启动Elasticsearch的同一节点上打开另一个命令shell窗口。<br>‘’[builder@master ~]$ curl -X GET “<a href="http://localhost:9200/_cat/health?v"" target="_blank" rel="noopener">http://localhost:9200/_cat/health?v"</a><br>‘’ epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent<br>‘’ 1547394013 15:40:13 elasticsearch green 1 1 0 0 0 0 0 0 - 100.0%<br>‘’<br>我们可以看到名为“elasticsearch”的群集处于green(绿色)状态。<br>每当我们要求群集健康时，我们要么获得绿色，黄色或红色。</p><blockquote><p>绿色 - 一切都很好（集群功能齐全）<br>黄色 - 所有数据均可用，但尚未分配一些副本（群集功能齐全）<br>红色 - 某些数据由于某种原因不可用（群集部分功能）<br><strong>注意：当群集为红色时，它将继续提供来自可用分片的搜索请求，但您可能需要尽快修复它，因为存在未分配的分片。</strong><br>同样从上面的响应中，我们可以看到总共1个节点，并且我们有0个分片，因为我们还没有数据。请注意，由于我们使用的是默认群集名称（elasticsearch），并且由于Elasticsearch默认使用单播网络发现来查找同一台计算机上的其他节点，因此您可能会意外启动计算机上的多个节点并将它们所有都加入一个集群。在这种情况下，您可能会在上面的响应中看到多个节点。<br>查看集群中的节点列表：<br>‘’[builder@master ]$ curl -X GET “<a href="http://localhost:9200/_cat/nodes?v"" target="_blank" rel="noopener">http://localhost:9200/_cat/nodes?v"</a><br>‘’ ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name<br>‘’ 127.0.0.1 20 98 8 1.90 mdi * siwsSwZ<br>从上面的结果中，我们可以看到一个名为“siwsSwZ”的节点，它是我们集群中当前的单个节点。</p></blockquote><h2 id="索引-indices-的CRUD、查询操作"><a href="#索引-indices-的CRUD、查询操作" class="headerlink" title="索引(indices)的CRUD、查询操作"></a>索引(indices)的CRUD、查询操作</h2><p>查看索引列表：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X GET <span class="string">"http://localhost:9200/_cat/indices?v"</span></span><br><span class="line">health status index uuid pri rep docs.count docs.deleted store.size pri.store.size</span><br></pre></td></tr></table></figure><p>因为我们的节点是刚刚建立的，还没数据，所以上面查询的结果只有一行字段名，没有索引数据</p><h3 id="创建索引"><a href="#创建索引" class="headerlink" title="创建索引"></a>创建索引</h3><p>现在让我们创建一个名为“customer”的索引，然后再次列出所有索引（下面第一个命令使用PUT动词创建名为“customer”的索引。加?pretty表示返回的结果打印格式化过的JSON（如果有的话）。）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X PUT <span class="string">"localhost:9200/customer?pretty"</span></span><br><span class="line"> {</span><br><span class="line"> <span class="string">"acknowledged"</span> : <span class="literal">true</span>,</span><br><span class="line"> <span class="string">"shards_acknowledged"</span> : <span class="literal">true</span>,</span><br><span class="line"> <span class="string">"index"</span> : <span class="string">"customer"</span></span><br><span class="line"> }</span><br><span class="line">[builder@master ~]$ curl -X GET <span class="string">"localhost:9200/_cat/indices?v"</span></span><br><span class="line">health status index uuid pri rep docs.count docs.deleted store.size pri.store.size</span><br><span class="line">yellow open customer i7R-XRH8R0Kn7uLQRNC_yQ 5 1 0 0 1.1kb 1.1kb</span><br></pre></td></tr></table></figure><p>第二个命令的结果告诉我们，我们现在有一个名为customer的索引，它有5个主分片和1个副本（默认值），并且它包含0个文档。<br>您可能还注意到客户索引标记了黄色运行状况。回想一下我们之前的讨论，黄色表示某些副本尚未（尚未）分配。此索引发生这种情况的原因是因为默认情况下Elasticsearch为此索引创建了一个副本。由于我们目前只有一个节点在运行，因此在另一个节点加入集群的较晚时间点之前，尚无法分配一个副本（用于高可用性）。将该副本分配到第二个节点后，此索引的运行状况将变为绿色。</p><p><em>Tips: 上面命令中，-X PUT 后的路径不要带http哟，否则会出现如下错误的：</em></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X PUT <span class="string">"http://localhost:9200/_cat/customer1?pretty"</span></span><br><span class="line"> {</span><br><span class="line"> <span class="string">"error"</span> : <span class="string">"Incorrect HTTP method for uri [/_cat/customer1?pretty] and method [PUT], allowed: [POST]"</span>,</span><br><span class="line"> <span class="string">"status"</span> : 405</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><h3 id="查询文档"><a href="#查询文档" class="headerlink" title="查询文档"></a>查询文档</h3><p>现在让我们在customer索引中加入一些内容。我们将一个简单的客户文档索引到客户索引中，ID为1，如下所示：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X PUT <span class="string">"localhost:9200/customer/_doc/1?pretty"</span> -H <span class="string">"Content-Type:application/json"</span> -d <span class="string">'{ "name": "Builder Luo"}'</span> </span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_index"</span> : <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"_version"</span> : 1,</span><br><span class="line"> <span class="string">"result"</span> : <span class="string">"created"</span>,</span><br><span class="line"> <span class="string">"_shards"</span> : {</span><br><span class="line"> <span class="string">"total"</span> : 2,</span><br><span class="line"> <span class="string">"successful"</span> : 1,</span><br><span class="line"> <span class="string">"failed"</span> : 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span> : 0,</span><br><span class="line"> <span class="string">"_primary_term"</span> : 2</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>curl命令说明：-X为请求方式，-H 参数为设置请求头，-d参数为请求参数 。<br>‘’ Tips: 关于：路径localhost:9200/customer/_doc/1?pretty<br>‘’ localhost:9200/customer是之前创建的索引，后接/_doc/1表示为在customer索引上创建1个ID为1的文档(?pretty表示将返回的响应json格式化美观)<br><strong>提示：值得注意的是，Elasticsearch在你将文档编入索引之前不需要先显式创建索引。在前面的示例中，如果customer索引事先尚未存在，则Elasticsearch将自动创建customer索引。</strong><br>查看刚刚创建好的文档:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X GET <span class="string">"localhost:9200/customer/_doc/1?pretty"</span></span><br><span class="line">{</span><br><span class="line"> <span class="string">"_index"</span> : <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"_version"</span> : 1,</span><br><span class="line"> <span class="string">"found"</span> : <span class="literal">true</span>,</span><br><span class="line"> <span class="string">"_source"</span> : {</span><br><span class="line"> <span class="string">"name"</span> : <span class="string">"Builder Luo"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="删除索引"><a href="#删除索引" class="headerlink" title="删除索引"></a>删除索引</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X DELETE <span class="string">"localhost:9200/customer?pretty"</span></span><br><span class="line">{</span><br><span class="line"> <span class="string">"acknowledged"</span> : <span class="literal">true</span></span><br><span class="line">}</span><br><span class="line">[builder@master ~]$ curl -X GET <span class="string">"localhost:9200/_cat/indices?v"</span></span><br><span class="line">health status index uuid pri rep docs.count docs.deleted store.size pri.store.size</span><br></pre></td></tr></table></figure><p>上面意味着索引已成功删除，我们现在回到我们在集群中没有任何内容的地方。<br>在我们继续之前，让我们再仔细看看到目前为止我们学到的一些API命令：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">curl -X PUT <span class="string">''</span>localhost:9200/customer<span class="string">''</span></span><br><span class="line">curl -X GET <span class="string">''</span>localhost:9200/customer<span class="string">''</span></span><br><span class="line">curl -X DELETE <span class="string">''</span>localhost:9200/customer<span class="string">''</span></span><br><span class="line"></span><br><span class="line">curl -X PUT <span class="string">''</span>localhost:9200/customer/_doc/1<span class="string">''</span> -H <span class="string">"Content-Type:application/json"</span> -d <span class="string">'{ "name": "Builder Luo"}'</span></span><br><span class="line">curl -X GET <span class="string">''</span>localhost:9200/customer/_doc/1<span class="string">''</span></span><br><span class="line">curl -X DELETE <span class="string">''</span>localhost:9200/customer/_doc/1<span class="string">''</span></span><br></pre></td></tr></table></figure><p>如果我们仔细研究上述命令，我们实际上可以看到我们如何在Elasticsearch中访问数据的模式。该模式可归纳如下：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><HTTP Verb> Node_address/<Index>/<Type>/<ID></span><br></pre></td></tr></table></figure><p>这种REST访问模式在所有API命令中都非常普遍，如果你能记住它，你将在掌握Elasticsearch方面有一个良好的开端。</p><h3 id="替换文档"><a href="#替换文档" class="headerlink" title="替换文档"></a>替换文档</h3><p>Elasticsearch几乎实时提供数据操作和搜索功能。默认情况下，从索引/更新/删除数据到搜索结果中显示的时间，您可能会有一秒钟的延迟（刷新间隔）。<br>数据在事务完成后立即可用，这是Elasticsearch与SQL等其他平台的重要区别<br>之前，我们执行过如下命令：</p><figure class="highlight 1c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">''[builder@master <span class="symbol">~]$ curl -X PUT "localhost</span>:<span class="number">9200</span>/customer/_doc/<span class="number">1</span>?pretty<span class="string">" -H "</span>Content-Type:application/json<span class="string">" -d '{ "</span>name<span class="string">": "</span>Builder Luo<span class="string">"}' </span></span><br><span class="line">下面，我们执行一样的命令，文档ID还是指定为<span class="number">1</span>，只是-d参数内容有变化：</span><br><span class="line">''[builder@master <span class="symbol">~]$ curl -X PUT "localhost</span>:<span class="number">9200</span>/customer/_doc/<span class="number">1</span>?pretty<span class="string">" -H "</span>Content-Type:application/json<span class="string">" -d '{ "</span>name<span class="string">": "</span>Baby Mon<span class="string">"}' </span></span><br><span class="line">''{</span><br><span class="line">'' <span class="string">"_index"</span> : <span class="string">"customer"</span>,</span><br><span class="line">'' <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line">'' <span class="string">"_id"</span> : <span class="string">"1"</span>,</span><br><span class="line">'' <span class="string">"_version"</span> : <span class="number">2</span>,</span><br><span class="line">'' <span class="string">"result"</span> : <span class="string">"updated"</span>,</span><br><span class="line">'' <span class="string">"_shards"</span> : {</span><br><span class="line">'' <span class="string">"total"</span> : <span class="number">2</span>,</span><br><span class="line">'' <span class="string">"successful"</span> : <span class="number">1</span>,</span><br><span class="line">'' <span class="string">"failed"</span> : <span class="number">0</span></span><br><span class="line">'' },</span><br><span class="line">'' <span class="string">"_seq_no"</span> : <span class="number">1</span>,</span><br><span class="line">'' <span class="string">"_primary_term"</span> : <span class="number">1</span></span><br><span class="line">'' }</span><br></pre></td></tr></table></figure><p>以上内容将ID为1的文档名称从“Builder Luo”更改为“Baby Mon”, result字段值为: updated。另一方面，如果我们使用不同的ID，则会对新文档编制索引，并且索引中已有的现有文档保持不变。<br><em>索引时，ID部分是可选的。如果未指定，Elasticsearch将生成随机ID，然后使用它来索引文档。Elasticsearch生成的实际ID（或前面示例中显式指定的内容）将作为索引API调用的一部分返回。</em><br>此示例显示如何在没有显式ID的情况下索引文档：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ curl -X POST <span class="string">"localhost:9200/customer/_doc?pretty"</span> -H <span class="string">"Content-Type:application/json"</span> -d <span class="string">'{ "name": "Moonlight Chen"}'</span></span><br><span class="line"> {</span><br><span class="line"> <span class="string">"_index"</span> : <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"UmGaVGgBol3Y-BIhbtd0"</span>,</span><br><span class="line"> <span class="string">"_version"</span> : 1,</span><br><span class="line"> <span class="string">"result"</span> : <span class="string">"created"</span>,</span><br><span class="line"> <span class="string">"_shards"</span> : {</span><br><span class="line"> <span class="string">"total"</span> : 2,</span><br><span class="line"> <span class="string">"successful"</span> : 1,</span><br><span class="line"> <span class="string">"failed"</span> : 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span> : 0,</span><br><span class="line"> <span class="string">"_primary_term"</span> : 1</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>可以看到上面自动生成的ID是：UmGaVGgBol3Y-BIhbtd0<br><strong>请注意，在上面的情况中，我们使用POST动词而不是PUT，因为我们没有指定ID。</strong></p><h3 id="修改文档数据"><a href="#修改文档数据" class="headerlink" title="修改文档数据"></a>修改文档数据</h3><p>除了能够重新索引和替换文档，我们还可以更新文档数据。（请注意，Elasticsearch实际上并没有在内部进行就地更新。每当我们进行更新时，Elasticsearch都会删除旧文档，然后一次性对应用了更新的新文档编制索引。）</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -d参数，需要封装成： { "doc": {dataJSON}}</span></span><br><span class="line">[builder@master ~]$ curl -X POST <span class="string">"localhost:9200/customer/_doc/1/_update?pretty"</span> -H <span class="string">"Content-Type:application/json"</span> -d <span class="string">'{ "doc": { "name": "My Baby Mon", "age": 20} }'</span> </span><br><span class="line">{</span><br><span class="line"><span class="string">"_index"</span> : <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"_version"</span> : 4,</span><br><span class="line"> <span class="string">"result"</span> : <span class="string">"updated"</span>,</span><br><span class="line"> <span class="string">"_shards"</span> : {</span><br><span class="line"> <span class="string">"total"</span> : 2,</span><br><span class="line"> <span class="string">"successful"</span> : 1,</span><br><span class="line"> <span class="string">"failed"</span> : 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span> : 3,</span><br><span class="line"> <span class="string">"_primary_term"</span> : 1</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 也可以使用简单脚本执行更新。此示例使用脚本将年龄增加5：</span></span><br><span class="line">[builder@master ~]$ curl -X POST <span class="string">"localhost:9200/customer/_doc/1/_update?pretty"</span> -H <span class="string">"Content-Type:application/json"</span> -d <span class="string">'{ "script": "ctx._source.age += 5" }'</span></span><br><span class="line">{</span><br><span class="line"> <span class="string">"_index"</span> : <span class="string">"customer"</span>,</span><br><span class="line"> <span class="string">"_type"</span> : <span class="string">"_doc"</span>,</span><br><span class="line"> <span class="string">"_id"</span> : <span class="string">"1"</span>,</span><br><span class="line"> <span class="string">"_version"</span> : 5,</span><br><span class="line"> <span class="string">"result"</span> : <span class="string">"updated"</span>,</span><br><span class="line"> <span class="string">"_shards"</span> : {</span><br><span class="line"> <span class="string">"total"</span> : 2,</span><br><span class="line"> <span class="string">"successful"</span> : 1,</span><br><span class="line"> <span class="string">"failed"</span> : 0</span><br><span class="line"> },</span><br><span class="line"> <span class="string">"_seq_no"</span> : 4,</span><br><span class="line"> <span class="string">"_primary_term"</span> : 1</span><br><span class="line"> }</span><br><span class="line"><span class="comment"># 在上面的示例中，ctx._source指的是即将更新的当前源文档。</span></span><br></pre></td></tr></table></figure><p>Elasticsearch提供了在给定查询条件（如SQL UPDATE-WHERE语句）的情况下更新多个文档的功能。请参阅[docs-update-by-queryAPI[<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docs-update-by-query.html]]" target="_blank" rel="noopener">https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docs-update-by-query.html]]</a><br>本文到这里，我们主要介绍了Elasticsearch的索引、文档的一些主要命令，我们将在下一篇文章中介绍Elasticsearch的批处理命令、以及探索操作数据的命令</p>]]></content>
<summary type="html">
<p><em>本教程是基于Elasticsearch6.5版本编写</em><br>在本系列教程上一篇文章中，我们介绍了Elasticsearch的一些基本概念、安装并运行起了一个Elasticsearch节点。现在我们已经启动并运行了节点（集群），下一步是了解如何与它进行通信。
</summary>
<category term="搜索引擎" scheme="https://blog.monbuilder.top/categories/%E6%90%9C%E7%B4%A2%E5%BC%95%E6%93%8E/"/>
<category term="Elasticsearch" scheme="https://blog.monbuilder.top/tags/Elasticsearch/"/>
</entry>
<entry>
<title>搭建大数据平台系列(4)-hive环境搭建</title>
<link href="https://blog.monbuilder.top/2019/03/06/hive/"/>
<id>https://blog.monbuilder.top/2019/03/06/hive/</id>
<published>2019-03-06T14:19:33.000Z</published>
<updated>2019-03-06T14:53:01.208Z</updated>
<content type="html"><![CDATA[<h2 id="0-准备步骤"><a href="#0-准备步骤" class="headerlink" title="0.准备步骤"></a>0.准备步骤</h2><p>Hive 是依赖在Hadoop上的，所以他的安装不需要像Hadoop或者spark那样每个节点都安装一遍，只需在Hadoop的master节点上安装一个即可。Hive的安装前，需要Hadoop的环境，以及Mysql。</p><h2 id="1-安装过程"><a href="#1-安装过程" class="headerlink" title="1.安装过程"></a>1.安装过程</h2><p>###1.1下载并解压安装包</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">下载hive-1.1.0-cdh5.5.0.tar.gz到master机器的~/bigdataspacce文件夹下</span></span><br><span class="line"><span class="meta">#</span><span class="bash">解压安装包的命令：</span></span><br><span class="line">[hadoop@master ~]$ cd ~/bigdataspacce</span><br><span class="line">[hadoop@master bigdataspace]$ tar -zxvf hive-1.1.0-cdh5.5.0.tar.gz</span><br><span class="line"><span class="meta">#</span><span class="bash">解压完成后删除压缩包：</span></span><br><span class="line">[hadoop@master bigdataspace]$ rm hive-1.1.0-cdh5.5.0.tar.gz</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">配置HIVE_HOME环境变量</span></span><br><span class="line">[hadoop@master ~]$ sudo vi /etc/profile</span><br><span class="line">(添加配置内容如下,红色为需要新增的配置)</span><br><span class="line">export HIVE_HOME=/home/hadoop/bigdataspace/hive-1.1.0-cdh5.5.0</span><br><span class="line">export PATH=$JAVA_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH</span><br><span class="line"><span class="meta">#</span><span class="bash">让环境变量生效</span></span><br><span class="line">[hadoop@master ~]$ source /etc/profile</span><br></pre></td></tr></table></figure><p>###1.2修改hive-env.sh配置文件</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ cd /home/hadoop/bigdataspace/hive-1.1.0-cdh5.5.0/conf</span><br><span class="line">[hadoop@master conf]$ cp hive-env.sh.template hive-env.sh</span><br><span class="line">[hadoop@master conf]$ vi hive-env.sh</span><br><span class="line"><span class="meta">#</span><span class="bash">在hive-env.sh配置文件末尾加上:</span></span><br><span class="line">export HADOOP_HOME=/home/hadoop/bigdataspace/hadoop-2.6.0-cdh5.5.0</span><br><span class="line">export HIVE_CONF_DIR=/home/hadoop/bigdataspace/hive-1.1.0-cdh5.5.0/conf</span><br></pre></td></tr></table></figure><p>###1.3新建hive-site.xml配置文件</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master conf]$ vi hive-env.sh</span><br><span class="line">##主要的配置内容如下：</span><br><span class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span><span class="meta">?></span></span></span><br><span class="line"><span class="php"><span class="meta"><?</span>xml-stylesheet type=<span class="string">"text/xsl"</span> href=<span class="string">"configuration.xsl"</span><span class="meta">?></span></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hive.metastore.warehouse.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/hive-1.1.0-cdh5.5.0/hive-db/warehouse<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>location of default database for the warehouse<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hive.exec.scratchdir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/hive-1.1.0-cdh5.5.0/tmp/hive-${user.name}<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Scratch space for Hive jobs<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hive.exec.local.scratchdir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/hive-1.1.0-cdh5.5.0/tmp/${user.name}<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Local scratch space for Hive jobs<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hive.downloaded.resources.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/hive-1.1.0-cdh5.5.0/downloaded<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span></span><br><span class="line">Temporary local directory for added resources in the remote file system.</span><br><span class="line"><span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hive.querylog.location<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/hive-1.1.0-cdh5.5.0/queryLogs/${user.name}<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Location of Hive run time structured log file<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>javax.jdo.option.ConnectionURL<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span></span><br><span class="line">jdbc:mysql://slave1:3306/hive?useUnicode=true&amp;characterEncoding=utf8</span><br><span class="line"><span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>JDBC connect string for a JDBC metastore<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>javax.jdo.option.ConnectionDriverName<span class="tag"></<span class="name">name</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>com.mysql.jdbc.Driver<span class="tag"></<span class="name">value</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Driver class name for a JDBC metastore<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>javax.jdo.option.ConnectionUserName<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hive<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>username to use against metastore database<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>javax.jdo.option.ConnectionPassword<span class="tag"></<span class="name">name</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hive<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>password to use against metastore database<span class="tag"></<span class="name">description</span>></span> </span><br><span class="line"><span class="tag"></<span class="name">property</span>></span> </span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><h3 id="1-4添加mysql-connector的jar包到hive安装路径下的lib文件夹"><a href="#1-4添加mysql-connector的jar包到hive安装路径下的lib文件夹" class="headerlink" title="1.4添加mysql-connector的jar包到hive安装路径下的lib文件夹"></a>1.4添加mysql-connector的jar包到hive安装路径下的lib文件夹</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"><span class="variable">$HIVE_HOME</span>为前面hive安装的目录路径:/home/hadoop/bigdataspace/hive-1.1.0-cdh5.5.0</span></span><br><span class="line">[hadoop@master ~] mv mysql-connector-java-5.1.33.jar $HIVE_HOME/lib</span><br></pre></td></tr></table></figure><h3 id="1-5启动元数据服务"><a href="#1-5启动元数据服务" class="headerlink" title="1.5启动元数据服务"></a>1.5启动元数据服务</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ cd ~/bigdataspace/hive-1.1.0-cdh5.5.0</span><br><span class="line">[hadoop@master hive-1.1.0-cdh5.5.0]$ ./bin/hive --service metastore &</span><br></pre></td></tr></table></figure><h3 id="1-6启动-停止hive-CTL-命令行"><a href="#1-6启动-停止hive-CTL-命令行" class="headerlink" title="1.6启动/停止hive (CTL)命令行"></a>1.6启动/停止hive (CTL)命令行</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">因为一开始配置了HIVE_HOME环境变量，可以直接在任何目录下执行hive命令了,进入hive控制台</span></span><br><span class="line">[hadoop@master bigdataspace]$ hive </span><br><span class="line">Logging initialized using configuration in jar:file:/home/hadoop/bigdataspace</span><br><span class="line">/hive-1.1.0-cdh5.5.0/lib/hive-common-1.1.0-cdh5.5.0.jar!/hive-log4j.properties</span><br><span class="line">WARNING: Hive CLI is deprecated and migration to Beeline is recommended.</span><br><span class="line">hive (default)></span><br></pre></td></tr></table></figure><p><em>上面报错了，解决Logging initialized using configuration in jar:file… （因为没log配置文件，直接从jar包查找）</em></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> <span class="built_in">cd</span> ~/bigdataspace/ /hive-1.1.0-cdh5.5.0/conf</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> cp beeline-log4j.properties.template beeline-log4j.properties</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> cp hive-log4j.properties.template hive-log4j.properties</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> cp hive-exec-log4j.properties.template hive-exec-log4j.properties</span></span><br><span class="line">[hadoop@master bigdataspace]$ hive</span><br><span class="line">Logging initialized using configuration in file:/home/hadoop/bigdataspace/</span><br><span class="line">hive-1.1.0-cdh5.5.0/conf/hive-log4j.properties</span><br><span class="line">WARNING: Hive CLI is deprecated and migration to Beeline is recommended.</span><br><span class="line">hive (default)></span><br><span class="line"></span><br><span class="line"><span class="meta">hive></span><span class="bash"> quit; <span class="comment">#(退出hive，使用exit也可以)</span></span></span><br></pre></td></tr></table></figure><h3 id="1-7启动-停止beeline命令行（CTL）"><a href="#1-7启动-停止beeline命令行（CTL）" class="headerlink" title="1.7启动/停止beeline命令行（CTL）"></a>1.7启动/停止beeline命令行（CTL）</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">启动：</span></span><br><span class="line">[hadoop@master bigdataspace]$ beeline</span><br><span class="line"><span class="meta">#</span><span class="bash">停止：</span></span><br><span class="line"><span class="meta">beeline></span><span class="bash"> !q</span></span><br></pre></td></tr></table></figure><h3 id="1-8HiveServer2的使用"><a href="#1-8HiveServer2的使用" class="headerlink" title="1.8HiveServer2的使用"></a>1.8HiveServer2的使用</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ cd ~/bigdataspace/hive-1.1.0-cdh5.5.0/bin/</span><br><span class="line">[hadoop@master bin]$ ./hiveserver2 & #后面的&表示改命名在系统后台执行</span><br><span class="line">(如果执行上面命令让界面无法回到命令行，可以按ctrl+C回到命令行，这里&会让hiverserver2在后台继续执行)</span><br><span class="line"><span class="meta">#</span><span class="bash">查看HiveServer2的进程情况(如果无则hiverserver2启动失败或停止了):</span></span><br><span class="line">[hadoop@master bin]$ ps -ef |grep HiveServer2 </span><br><span class="line">hadoop 25545 14762 3 17:02 pts/1 00:00:21 /home/hadoop/bigdataspace/jdk1.8.0_60/bin/java -Xmx256m -Djava.library.path=/home/hadoop/bigdataspace/hadoop-2.6.0-cdh5.5.0/lib/native/ -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/bigdataspace/hadoop-2.6.0-cdh5.5.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/bigdataspace/hadoop-2.6.0-cdh5.5.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /home/hadoop/bigdataspace/hive-1.1.0-cdh5.5.0/lib/hive-service-1.1.0-cdh5.5.0.jar org.apache.hive.service.server.HiveServer2</span><br><span class="line">hadoop 26038 14762 0 17:14 pts/1 00:00:00 grep HiveServer2</span><br><span class="line"></span><br><span class="line">(“kill -9 PID” 可以通过kill停止hiveserver2的后台服务) </span><br><span class="line"></span><br><span class="line">使用beeline连接hiveserver2测试：</span><br><span class="line">(</span><br><span class="line">jdbc:hive2：表示连接到hiveserver2</span><br><span class="line">master:表示hiveserver2安装的机器host/IP</span><br><span class="line">10001:表示hiveserver2设置的端口号(hive-site.xml中可设置)</span><br><span class="line">)</span><br><span class="line">[hadoop@master hive-1.1.0-cdh5.5.0]$ beeline -u jdbc:hive2://master:10001</span><br><span class="line"><span class="meta">#</span><span class="bash"><span class="comment">##这里可能会出现一些slf4j包有多个，引用异常，但是不是报错，如：</span></span></span><br><span class="line">SLF4J: Class path contains multiple SLF4J bindings</span><br><span class="line">SLF4J: Found binding in [jar:file:/home/hadoop/bigdataspace/had…)</span><br><span class="line">Connecting to jdbc:hive2://master:10001</span><br><span class="line">Connected to: Apache Hive (version 1.1.0-cdh5.5.0)</span><br><span class="line">Driver: Hive JDBC (version 1.1.0-cdh5.5.0)</span><br><span class="line">Transaction isolation: TRANSACTION_REPEATABLE_READ</span><br><span class="line">Beeline version 1.1.0-cdh5.5.0 by Apache Hive</span><br><span class="line">0: jdbc:hive2://master:10001></span><br></pre></td></tr></table></figure><p>以上完成了Hive的基本安装配置。</p>]]></content>
<summary type="html">
<h2 id="0-准备步骤"><a href="#0-准备步骤" class="headerlink" title="0.准备步骤"></a>0.准备步骤</h2><p>Hive 是依赖在Hadoop上的，所以他的安装不需要像Hadoop或者spark那样每个节点都安装一遍，只
</summary>
<category term="大数据" scheme="https://blog.monbuilder.top/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Hadoop" scheme="https://blog.monbuilder.top/tags/Hadoop/"/>
</entry>
<entry>
<title>Elasticsearch学习教程系列(0)-入门与安装</title>
<link href="https://blog.monbuilder.top/2019/01/10/elasticsearch-learn00/"/>
<id>https://blog.monbuilder.top/2019/01/10/elasticsearch-learn00/</id>
<published>2019-01-10T14:34:05.000Z</published>
<updated>2019-03-06T14:23:56.929Z</updated>
<content type="html"><![CDATA[<p><em>本教程是基于Elasticsearch6.5版本编写</em><br>下面将介绍、安装并启动Elasticsearch，查看其中的内容以及执行索引，搜索和修改数据等基本操作的过程。在本教程结束时，您应该很好地了解Elasticsearch是什么，它是如何工作的，并希望能够获得灵感，看看如何使用它来构建复杂的搜索应用程序或从数据中挖掘智能。</p><h2 id="0-1入门简介"><a href="#0-1入门简介" class="headerlink" title="0.1入门简介"></a>0.1入门简介</h2><p>Elasticsearch是一个高度可扩展的开源全文搜索和分析引擎。Elasticsearch基于Apache Lucene，它允许您快速，近实时地存储，搜索和分析大量数据。它通常用作与底层引擎/技术，提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口；为具有复杂搜索功能和要求的应用程序提供支持。<br>以下是Elasticsearch可用于的一些场景示例：</p><ul><li>您运行在线网上商店，允许您的客户搜索您销售的产品。在这种情况下，您可以使用Elasticsearch存储整个产品目录和库存，并为它们提供搜索和自动填充建议。</li><li>您希望收集日志或交易数据，并且希望分析和挖掘此数据以查找趋势，统计信息，摘要或异常。在这种情况下，您可以使用Logstash（Elasticsearch / Logstash / Kibana堆栈的一部分）来收集，聚合和解析数据，然后让Logstash将此数据提供给Elasticsearch。一旦数据在Elasticsearch中，您就可以运行搜索和聚合来挖掘您感兴趣的任何信息。</li><li>您运行价格警报平台，允许精通价格的客户指定一条规则，例如“我有兴趣购买特定的电子产品，如果小工具的价格在下个月内从任何供应商降至X美元以下，我希望收到通知” 。在这种情况下，您可以刮取供应商价格，将其推入Elasticsearch并使用其反向搜索（Percolator）功能来匹配价格变动与客户查询，并最终在发现匹配后将警报推送给客户。</li><li>您有分析/业务智能需求，并希望快速调查，分析，可视化并询问有关大量数据的特定问题（想想数百万或数十亿条记录）。在这种情况下，您可以使用Elasticsearch存储数据，然后使用Kibana（Elasticsearch / Logstash / Kibana堆栈的一部分）构建自定义仪表板，以便可视化对您来说重要的数据方面。此外，您可以使用Elasticsearch聚合功能针对您的数据执行复杂的商业智能查询。<h2 id="0-2基本概念"><a href="#0-2基本概念" class="headerlink" title="0.2基本概念"></a>0.2基本概念</h2><strong>有一些概念是Elasticsearch的核心。从一开始就理解这些概念将极大地帮助简化学习过程。</strong><br><em>1.近实时(NRT)</em><br>Elasticsearch是一个近实时搜索平台。这意味着从索引文档到可搜索文档的时间有一点延迟（通常是一秒）。<br><em>2.集群(cluster)</em><br>集群是一个或多个节点（服务器）的集合，它们共同保存您的整个数据，并提供跨所有节点的联合索引和搜索功能。群集由唯一名称标识，默认情况下为“elasticsearch”。此名称很重要，因为如果节点设置为按名称加入群集，则该节点只能是群集的一部分。确保不要在不同的环境中重用相同的群集名称，否则最终会导致节点加入错误的群集。例如，您可以使用logging-dev，logging-stage以及logging-prod 用于开发，登台和生产集群。请注意，如果群集中只有一个节点，那么它是完全正常的。此外，您还可以拥有多个独立的集群，每个集群都有自己唯一的集群名称。<br><em>3.节点(node)</em><br>节点是作为群集一部分的单个服务器，存储数据并参与群集的索引和搜索功能。就像集群一样，节点由名称标识，默认情况下，该名称是在启动时分配给节点的随机通用唯一标识符（UUID）。如果不需要默认值，可以定义所需的任何节点名称。此名称对于管理目的非常重要，您可以在其中识别网络中哪些服务器与Elasticsearch集群中的哪些节点相对应。可以将节点配置为按群集名称加入特定群集。默认情况下，每个节点都设置为加入一个名为cluster的集群elasticsearch，这意味着如果您在网络上启动了许多节点并且假设它们可以相互发现 - 它们将自动形成并加入一个名为的集群elasticsearch。在单个群集中，您可以拥有任意数量的节点。此外，如果您的网络上当前没有其他Elasticsearch节点正在运行，则默认情况下启动单个节点将形成一个名为的新单节点集群elasticsearch。<br><em>4.索引(index)</em><br>索引是具有某些类似特征的文档集合。例如，您可以拥有客户数据的索引，产品目录的另一个索引以及订单数据的另一个索引。索引由名称标识（必须全部小写），此名称用于在对其中的文档执行索引，搜索，更新和删除操作时引用索引。在单个群集中，您可以根据需要定义任意数量的索引。<br><em>5.文档(document)</em><br>文档是可以编制索引的基本信息单元。例如，您可以为单个客户提供文档，为单个产品提供另一个文档，为单个订单提供另一个文档。该文档以JSON（JavaScript Object Notation）表示，JSON是一种普遍存在的互联网数据交换格式。在索引/类型中，您可以根据需要存储任意数量的文档。请注意，尽管文档实际上驻留在索引中，但实际上必须将文档编入索引/分配给索引中的类型。<br><em>6.分片和副本(Shards & Replicas)</em><br>索引可能存储大量可能超过单个节点的硬件限制的数据。例如，占用1TB磁盘空间的十亿个文档的单个索引可能不适合单个节点的磁盘，或者可能太慢而无法单独从单个节点提供搜索请求。为了解决这个问题，Elasticsearch提供了将索引细分为多个称为分片的功能。创建索引时，只需定义所需的分片数即可。每个分片本身都是一个功能齐全且独立的“索引”，可以托管在集群中的任何节点上。<br>分片很重要，主要有两个原因：</li><li>它允许您水平拆分/缩放内容量</li><li>它允许您跨分片（可能在多个节点上）分布和并行化操作，从而提高性能/吞吐量<br>分片的分布方式以及如何将其文档聚合回搜索请求的机制完全由Elasticsearch管理，对用户而言是透明的。在可以随时发生故障的网络/云环境中，非常有用，强烈建议使用故障转移机制，以防分片/节点以某种方式脱机或因任何原因消失。为此，Elasticsearch允许您将索引的分片的一个或多个副本制作成所谓的副本分片或简称副本。<br>复制很重要，主要有两个原因：</li><li>它在碎片/节点发生故障时提供高可用性。因此，请务必注意，副本分片永远不会在与从中复制的原始/主分片相同的节点上分配。</li><li>它允许您扩展搜索量/吞吐量，因为可以在所有副本上并行执行搜索。<br>总而言之，每个索引可以拆分为多个分片。索引也可以复制为零（表示没有副本）或更多次。复制后，每个索引都将具有主分片（从中复制的原始分片）和副本分片（主分片的副本）。可以在创建索引时为每个索引定义分片和副本的数量。创建索引后，您还可以随时动态更改副本数。您可以使用_shrink和_splitAPI 更改现有索引的分片数，但这不是一项简单的任务，并且预先计划正确数量的分片是最佳方法。默认情况下，Elasticsearch中的每个索引都分配了5个主分片和1个副本，这意味着如果群集中至少有两个节点，则索引将包含5个主分片和另外5个副本分片（1个完整副本），总计为每个索引10个分片。<br><code>Tips: 每个Elasticsearch分片都是Lucene索引。单个Lucene索引中可以包含最大数量的文档。截止LUCENE-5843，限制是2,147,483,519（= Integer.MAX_VALUE - 128）文档。您可以使用_cat/shardsAPI 监控分片大小。</code><h2 id="0-3安装"><a href="#0-3安装" class="headerlink" title="0.3安装"></a>0.3安装</h2>Elasticsearch是基于Java开发的，因此安装时，需要JDK环境。我们安装的版本是Elasticsearch6.5.4，至少需要JDK 8。我们假设您已经在您的Linux(例如CentOS6.5)环境下已经安装了JDK。</li></ul><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">[builder@master ~]$ java -version</span><br><span class="line">java version "1.8.0_171"</span><br><span class="line">Java(TM) SE Runtime Environment (build 1.8.0_171-b11)</span><br><span class="line">Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">下载安装包，并解压：</span></span><br><span class="line">[builder@master ~]$ cd ~</span><br><span class="line">[builder@master ~]$ mdkir env</span><br><span class="line">[builder@master ~]$ cd env</span><br><span class="line">[builder@master ~]$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.tar.gz</span><br><span class="line"><span class="meta">#</span><span class="bash">等待下载完成后，解压</span></span><br><span class="line">[builder@master ~]$ tar -zxvf elasticsearch-6.5.4.tar.gz</span><br><span class="line">[builder@master ~]$ cd elasticsearch-6.5.4</span><br><span class="line"><span class="meta">#</span><span class="bash"> 在elasticsearch命令后加 -d 参数，是指以守护线程启动程序，即是后台运行</span></span><br><span class="line">[builder@master ~]$ ./bin/elasticsearch -d</span><br><span class="line"></span><br><span class="line">到这里Elasticsearch的安装已经完成了，是不是非常简单呢？</span><br><span class="line">另外，我们启动的时候是可以通过 -E 参数指定集群名称(cluster.name)或者节点名称(node.name)的：</span><br><span class="line">[builder@master ~]$ ./bin/elasticsearch -Ecluster.name=esCluster -Enode.name=builder01 -d</span><br></pre></td></tr></table></figure><p><code>Tips: 默认情况下，Elasticsearch使用port 9200来提供对其REST API的访问。如有必要，可以配置此端口。</code></p>]]></content>
<summary type="html">
<p><em>本教程是基于Elasticsearch6.5版本编写</em><br>下面将介绍、安装并启动Elasticsearch，查看其中的内容以及执行索引，搜索和修改数据等基本操作的过程。在本教程结束时，您应该很好地了解Elasticsearch是什么，它是如何工作的，并希
</summary>
<category term="搜索引擎" scheme="https://blog.monbuilder.top/categories/%E6%90%9C%E7%B4%A2%E5%BC%95%E6%93%8E/"/>
<category term="Elasticsearch" scheme="https://blog.monbuilder.top/tags/Elasticsearch/"/>
</entry>
<entry>
<title>搭建大数据平台系列(3)-hbase环境搭建</title>
<link href="https://blog.monbuilder.top/2018/12/13/hbase/"/>
<id>https://blog.monbuilder.top/2018/12/13/hbase/</id>
<published>2018-12-13T09:21:59.000Z</published>
<updated>2018-12-14T06:14:20.392Z</updated>
<content type="html"><![CDATA[<h2 id="1-安装步骤"><a href="#1-安装步骤" class="headerlink" title="1.安装步骤"></a>1.安装步骤</h2><p>Hbase的安装需要在hadoop和zookeeper安装完成的基础上进行安装部署，所以，需要在安装hbase前准备好hadoop和zookeeper的环境（请看本系列前几篇文章）<br><figure class="highlight markdown"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">`1.下载hbase-1.0.0-cdh5.5.0.tar.gz到master机器的/bigdataspacce文件夹下</span><br><span class="line">2.解压安装包的命令：</span><br><span class="line">[<span class="string">hadoop@master </span>](<span class="link"></span>)$ cd /bigdataspacce</span><br><span class="line">[<span class="string">hadoop@master bigdataspace</span>](<span class="link"></span>)$ tar -zxvf hbase-1.0.0-cdh5.5.0.tar.gz</span><br><span class="line">3.解压完成后删除压缩包：</span><br><span class="line">[<span class="string">hadoop@master bigdataspace</span>](<span class="link"></span>)$ rm hbase-1.0.0-cdh5.5.0.tar.gz</span><br><span class="line">4.修改hbase-env.sh、hbase-site.xml配置文件以及regionservers文件（配置dataNode节点）</span><br><span class="line">$ cd /home/hadoop/bigdataspace/hbase-1.0.0-cdh5.5.0/conf</span><br><span class="line">$ vi hbase-env.sh</span><br><span class="line"><span class="section"># The java implementation to use. Java 1.7+ required.</span></span><br><span class="line"><span class="section"># export JAVA_HOME=/usr/java/jdk1.6.0/</span></span><br><span class="line">(在上面这条注释下加上：)</span><br><span class="line">export JAVA<span class="emphasis">_HOME=/home/hadoop/bigdataspace/jdk1.8.0_</span>60</span><br><span class="line">……</span><br><span class="line">export HBASE<span class="emphasis">_PID_</span>DIR=/data/hbase-1.0.0-cdh5.5.0/pids</span><br><span class="line"><span class="section"># export HBASE_MANAGES_ZK=true #设置hbase是否管理zookeeper</span></span><br><span class="line">export HBASE<span class="emphasis">_MANAGES_</span>ZK=true</span><br><span class="line">export HBASE<span class="emphasis">_MANAGES_</span>ZK=false #使用独立的ZooKeeper时需要修改HBASE<span class="emphasis">_MANAGES_</span>ZK值为false，为不使用默认自带的ZooKeeper实例。</span><br></pre></td></tr></table></figure></p><figure class="highlight taggerscript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">`$ vi hbase-site.xml</span><br><span class="line">(修改配置文件内容为如下)</span><br><span class="line"><span class="symbol">\<</span>configuration<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>name<span class="symbol">\></span>hbase.rootdir<span class="symbol">\<</span>/name<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>value<span class="symbol">\></span>hdfs://master:8020/hbase<span class="symbol">\<</span>/value<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>/property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>name<span class="symbol">\></span>hbase.cluster.distributed<span class="symbol">\<</span>/name<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>value<span class="symbol">\></span>true<span class="symbol">\<</span>/value<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>/property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>name<span class="symbol">\></span>hbase.zookeeper.quorum<span class="symbol">\<</span>/name<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>value<span class="symbol">\></span>slave1,slave2,slave3<span class="symbol">\<</span>/value<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>/property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>name<span class="symbol">\></span>hbase.zookeeper.property.dataDir<span class="symbol">\<</span>/name<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>value<span class="symbol">\></span>/data/zookeeper-3.4.5-cdh5.5.0/var/data<span class="symbol">\<</span>/value<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>/property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>name<span class="symbol">\></span>hbase.tmp.dir<span class="symbol">\<</span>/name<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>value<span class="symbol">\></span>/data/hbase-1.0.0-cdh5.5.0/tmp<span class="symbol">\<</span>/value<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>/property<span class="symbol">\></span></span><br><span class="line"><span class="symbol">\<</span>/configuration<span class="symbol">\></span></span><br><span class="line">(hdfs://master:8020/hbase,这里的hbase目录未建好的话是需要hdfs dfs –mkdir 新建的目录)</span><br><span class="line">$ vi regionservers</span><br><span class="line">slave1</span><br><span class="line">slave2</span><br><span class="line">slave3</span><br><span class="line">(以上使用对应的ip配置也可以)</span><br><span class="line">```</span><br></pre></td></tr></table></figure><p>`5.配置HBASE_HOME<br>$ vi /etc/profile<br>(加上如下配置)<br>export HBASE_HOME=/home/hadoop/bigdataspace/hbase-1.0.0-cdh5.5.0<br>export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$PATH<br>6.使用scp命令把hbase分发到各个节点<br>$ scp -r hbase-1.0.0-cdh5.5.0/ hadoop@slave1:/bigdataspace/<br>$ scp -r hbase-1.0.0-cdh5.5.0/ hadoop@slave2:/bigdataspace/<br>$ scp -r hbase-1.0.0-cdh5.5.0/ hadoop@slave3:/bigdataspace/</p><p>然后在各个节点上执行第5步：配置HBASE_HOME_<br><figure class="highlight clean"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">`</span><br></pre></td></tr></table></figure></p><p>`7.Hbase的启动与停止<br>启动hbase时要确保hdfs已经启动，HBase的启动顺序为：HDFS->Zookeeper->HBase，启动Hbase的命令如下(在master机器上)：<br><a href="">hadoop@master </a>$ cd /home/hadoop/bigdataspace/hbase-1.0.0-cdh5.5.0/bin<br>(注意，如果设置了hbase管理zookeeper，则需要先关闭手动启动的各节点zookeeper)<br>如slave1机器：<br><a href="">hadoop@slave1 </a>$ /bigdataspace/zookeeper-3.4.5-cdh5.5.0/bin/zkServer.sh stop<br>在master机器：<br> <a href="">hadoop@master bin</a>$ ./start-hbase.sh</p><p><a href="">hadoop@master bin</a>$ jps<br>29385 HMaster<br>19994 JobHistoryServer<br>19068 NameNode<br>29757 Jps<br>19422 ResourceManager<br>19263 SecondaryNameNode<br><figure class="highlight clean"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">`</span><br></pre></td></tr></table></figure></p><p>`如slave1机器：<br><a href="">hadoop@slave1 bin</a>$ jps<br>12768 DataNode<br>17971 HRegionServer<br>12884 NodeManager<br>22704 QuorumPeerMain<br>18169 Jps<br>17851 HQuorumPeer #hbase管理zookeeper的进程, </p><h1 id="如果export-HBASE-MANAGES-ZK-true，才会出现上面的进程"><a href="#如果export-HBASE-MANAGES-ZK-true，才会出现上面的进程" class="headerlink" title="如果export HBASE_MANAGES_ZK=true，才会出现上面的进程"></a>如果export HBASE_MANAGES_ZK=true，才会出现上面的进程</h1><p>如果HQuorumPeer不存在，而是QuorumPeerMain则表明需要手动关闭zookeeper，hbase才能接手管理。<br>Hbase停止命令：<br><a href="">hadoop@master bin</a>$ ./stop-hbase.sh<br><code>`</code><br>`## 2.验证启动成功<br><em>访问HBase web 页面：<a href="http://master:60010/" target="_blank" rel="noopener">http://master:60010/</a></em></p>]]></content>
<summary type="html">
<h2 id="1-安装步骤"><a href="#1-安装步骤" class="headerlink" title="1.安装步骤"></a>1.安装步骤</h2><p>Hbase的安装需要在hadoop和zookeeper安装完成的基础上进行安装部署，所以，需要在安装hbas
</summary>
<category term="大数据" scheme="https://blog.monbuilder.top/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Hadoop" scheme="https://blog.monbuilder.top/tags/Hadoop/"/>
</entry>
<entry>
<title>搭建大数据平台系列(2)-zookeeper环境搭建</title>
<link href="https://blog.monbuilder.top/2018/11/05/zookeeper/"/>
<id>https://blog.monbuilder.top/2018/11/05/zookeeper/</id>
<published>2018-11-05T08:08:10.000Z</published>
<updated>2018-12-14T06:14:20.400Z</updated>
<content type="html"><![CDATA[<h2 id="1-安装步骤"><a href="#1-安装步骤" class="headerlink" title="1.安装步骤"></a>1.安装步骤</h2><p>Zookeeper集群一般配置奇数个，在本次测试机是部署到slave1，slave2，slave3这3台机器上。<br><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">1</span>.下载zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>.tar.gz到slave1机器的~/bigdataspacce文件夹下</span><br><span class="line"><span class="number">2</span>.解压安装包的命令：</span><br><span class="line">[hadoop@slave1 bigdataspace]$ tar -zxvf zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>.tar.gz</span><br><span class="line"><span class="number">3</span>.解压完成后删除压缩包：</span><br><span class="line">[hadoop@slave1 bigdataspace]$ rm zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>.tar.gz</span><br><span class="line"><span class="number">4</span>.配置zoo.cfg文件</span><br><span class="line"> $ cd /home/hadoop/bigdataspace/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/conf</span><br><span class="line"> $ cp zoo_sample.cfg zoo.cfg</span><br><span class="line"> $ vi zoo.cfg</span><br><span class="line"> (修改文件中的dataDir配置)</span><br><span class="line"> dataDir=/data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/data</span><br><span class="line"> dataLogDir=/data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/dataLog/</span><br><span class="line">（并在clientPort下面新增如下配置）</span><br><span class="line">server<span class="meta">.1</span>=slave1:<span class="number">2888</span>:<span class="number">3888</span></span><br><span class="line">server<span class="meta">.2</span>=slave2:<span class="number">2888</span>:<span class="number">3888</span></span><br><span class="line">server<span class="meta">.3</span>=slave3:<span class="number">2888</span>:<span class="number">3888</span></span><br><span class="line"><span class="number">5</span>.建立dataDir对应路径的文件夹，并进入该data文件夹下新建一个文件myid：</span><br><span class="line"> $ mkdir -p /data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/dataLog</span><br><span class="line">$ mkdir -p /data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/data</span><br><span class="line"> $ cd /data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/data</span><br><span class="line"> $ vi myid</span><br><span class="line"> (myid文件内容为zoo.cfg中配的server号码，如server<span class="meta">.1</span>则myid文件中只保存<span class="number">1</span>，每台机器都配自己对应的号码)</span><br><span class="line"> $ cat myid</span><br><span class="line"> <span class="number">1</span></span><br><span class="line"> $</span><br><span class="line"><span class="number">6</span>.以上对zookeeper的配置基本完成，下面使用scp把zookeeper发到各个节点：</span><br><span class="line">$ scp -r zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/ hadoop@slave2:~/bigdataspace/</span><br><span class="line">$ scp -r zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/ hadoop@slave3:~/bigdataspace/</span><br><span class="line"><span class="number">7</span>.通过scp把myid传到各个节点，并修改其zoo.cfg配置文件对应的server号码</span><br><span class="line">(如server<span class="meta">.2</span>=slave1:<span class="number">52888</span>:<span class="number">53888</span>则myid文件存入<span class="number">2</span>)</span><br><span class="line"> $ scp -r /data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/ hadoop@slave2:/data/</span><br><span class="line"> $ scp -r /data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/ hadoop@slave3:/data/</span><br><span class="line"> (然后到到各个节点上修改/data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/data/myid文件),如：</span><br><span class="line"> [hadoop@slave2 ~]$ vi /data/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/var/data/myid</span><br><span class="line"> <span class="number">2</span></span><br><span class="line"> [hadoop@slave2 ~]$</span><br><span class="line"><span class="number">8.</span>zookeeper.out以及log4j日志文件的设置</span><br><span class="line">[hadoop@slave1 ~]$ cd /home/hadoop/bigdataspace/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/conf</span><br><span class="line">[hadoop@slave1 conf]$ vi log4j.properties</span><br><span class="line"># Define some <span class="meta">default</span> values that can be overridden by system properties</span><br><span class="line">zookeeper.root.logger=INFO, ROLLINGFILE</span><br><span class="line">……</span><br><span class="line">log4j.appender.ROLLINGFILE=org.apache.log4j.DailyRollingFileAppender</span><br><span class="line">查看zkServer.sh脚本，发现运行时会先加载zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/libexec/zkEnv.sh，不存在会加载zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/bin/zkEnv.sh</span><br><span class="line">[hadoop@slave1 ~]$ cd /home/hadoop/bigdataspace/zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/libexec</span><br><span class="line">[hadoop@slave1 libexec]$ vi zkEnv.sh</span><br><span class="line">if [ <span class="string">"x${ZOO_LOG_DIR}"</span> = <span class="string">"x"</span> ]</span><br><span class="line">then</span><br><span class="line"> ZOO_LOG_DIR=<span class="string">"/data/zookeeper-3.4.5-cdh5.5.0/logs"</span></span><br><span class="line">fi</span><br><span class="line">if [ <span class="string">"x${ZOO_LOG4J_PROP}"</span> = <span class="string">"x"</span> ]</span><br><span class="line">then</span><br><span class="line">ZOO_LOG4J_PROP=<span class="string">"INFO,ROLLINGFILE"</span></span><br><span class="line">也可以把zookeeper-<span class="number">3.4</span><span class="meta">.5</span>-cdh5<span class="meta">.5</span><span class="meta">.0</span>/bin/zkEnv.sh文件的配置修改成上面一样.</span><br></pre></td></tr></table></figure></p><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">9</span>.启动zookeeper服务：</span><br><span class="line"> [hadoop<span class="variable">@slave1</span> ~]<span class="variable">$ </span>cd /home/hadoop/bigdataspace/zookeeper-<span class="number">3.4</span>.<span class="number">5</span>-cdh5.<span class="number">5.0</span>/bin</span><br><span class="line"> [hadoop<span class="variable">@slave1</span> bin]<span class="variable">$ </span>./ zkServer.sh start <span class="comment">#启动zookeeper（每台机器都要执行此命令）</span></span><br></pre></td></tr></table></figure><h2 id="2-验证"><a href="#2-验证" class="headerlink" title="2.验证"></a>2.验证</h2><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"> [hadoop@slave1 bin]$ jps #使用jps命令</span><br><span class="line"> <span class="number">25906</span> Jps</span><br><span class="line"><span class="number">20536</span> QuorumPeerMain #zookeeper的进程</span><br><span class="line"><span class="number">19994</span> JobHistoryServer</span><br><span class="line"><span class="number">19068</span> NameNode</span><br><span class="line"><span class="number">19422</span> ResourceManager</span><br><span class="line"><span class="number">19263</span> SecondaryNameNode</span><br></pre></td></tr></table></figure><p>如上，含有QuorumPeerMain表明安装成功<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master bin]$ ./ zkServer.sh stop #停止zookeeper（每台机器都要执行此命令）</span><br><span class="line">[hadoop@master bin]$ ./zkServer.sh status #查看角色状态命令</span><br><span class="line">JMX enabled by default</span><br><span class="line">Using config: /home/hadoop/bigdataspace/zookeeper-3.4.5-cdh5.5.0/bin/<span class="built_in">..</span>/conf/zoo.cfg</span><br><span class="line">Mode: follower</span><br><span class="line">（Mode: follower/leader，leader这个角色只有一台机器，是通过zookeeper的选举算法产生）</span><br><span class="line">如果出现如下错误，</span><br><span class="line">[hadoop@master bin]$ ./zkServer.sh status</span><br><span class="line">JMX enabled by default</span><br><span class="line">Using config: /home/hadoop/bigdataspace/zookeeper-3.4.5-cdh5.5.0/bin/<span class="built_in">..</span>/conf/zoo.cfg</span><br><span class="line"><span class="builtin-name">Error</span> contacting service. It is probably <span class="keyword">not</span> running.</span><br><span class="line">极大可能是因为防火墙端口被限制了，可以打开这些被用到的端口</span><br><span class="line">(注意：只启用一台zookeeper也是会 出现这个错误，需要启动2台以上的节点)</span><br></pre></td></tr></table></figure></p><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#进入zookeeper客户端的命令</span></span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bin</span>]$ bin/zkCli.sh -server <span class="literal">master</span>:<span class="number">2181</span></span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h2 id="1-安装步骤"><a href="#1-安装步骤" class="headerlink" title="1.安装步骤"></a>1.安装步骤</h2><p>Zookeeper集群一般配置奇数个，在本次测试机是部署到slave1，slave2，slave3这3台机器
</summary>
<category term="大数据" scheme="https://blog.monbuilder.top/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Hadoop" scheme="https://blog.monbuilder.top/tags/Hadoop/"/>
</entry>
<entry>
<title>rabbitMQ的介绍与安装使用</title>
<link href="https://blog.monbuilder.top/2018/10/24/rabbitmq/"/>
<id>https://blog.monbuilder.top/2018/10/24/rabbitmq/</id>
<published>2018-10-24T01:47:37.000Z</published>
<updated>2018-12-14T06:14:20.387Z</updated>
<content type="html"><![CDATA[<h2 id="0-消息中间件的介绍"><a href="#0-消息中间件的介绍" class="headerlink" title="0. 消息中间件的介绍"></a>0. 消息中间件的介绍</h2><p>RabbitMQ是一个“传统”消息代理，可以实现各种消息传递协议。它是首批实现合理级别功能，客户端库，开发工具和质量文档的开源消息代理之一。RabbitMQ最初是为实现AMQP而开发的，AMQP是一种开放式线路协议，具有强大的路由功能。虽然Java具有像JMS这样的消息传递标准，但它对于需要分布式消息传递的非Java应用程序没有帮助，因为它严重限制了任何集成场景，微服务或单片机。随着AMQP的出现，跨语言的灵活性成为开源消息代理的真实存在。<br>RabbitMQ被设计为通用消息代理，采用点对点，请求/回复和pub-sub通信样式模式的多种变体。它使用智能代理/哑消费者模型，专注于向消费者提供一致的消息传递，消费者的消费速度与经纪人跟踪消费者状态的速度大致相似。它是成熟的，在正确配置时表现良好，得到很好的支持（客户端库Java，.NET，node.js，Ruby，PHP和更多语言），并且有许多可用的插件可以将它扩展到更多的用例和集成场景。</p><h2 id="1-kafka-rabbitMQ的对比"><a href="#1-kafka-rabbitMQ的对比" class="headerlink" title="1.kafka,rabbitMQ的对比"></a>1.kafka,rabbitMQ的对比</h2><p>RabbitMQ中的通信可以根据需要同步或异步。发布者向交换发送消息，消费者从队列中检索消息。通过交换将生产者与队列分离可确保生产者不会受到硬编码路由决策的影响。RabbitMQ还提供了许多分布式部署方案（并且确实要求所有节点都能够解析主机名）。可以将多节点群集设置为群集联合，并且不依赖于外部服务（但某些群集形成插件可以使用AWS API，DNS，Consul等）。<br>Apache Kafka专为高容量发布 - 订阅消息和流而设计，旨在持久，快速和可扩展。从本质上讲，Kafka提供了一个持久的消息存储，类似于日志，在服务器集群中运行，它存储称为主题的类别中的记录流。</p><h2 id="2-rabbitMQ的安装与使用"><a href="#2-rabbitMQ的安装与使用" class="headerlink" title="2.rabbitMQ的安装与使用"></a>2.rabbitMQ的安装与使用</h2><h3 id="2-1在基于RPM的Linux上安装（RHEL，CentOS，Fedora，openSUSE），RabbitMQ-RPM包需要sudo权限-或使用root用户-才能安装和管理。"><a href="#2-1在基于RPM的Linux上安装（RHEL，CentOS，Fedora，openSUSE），RabbitMQ-RPM包需要sudo权限-或使用root用户-才能安装和管理。" class="headerlink" title="2.1在基于RPM的Linux上安装（RHEL，CentOS，Fedora，openSUSE），RabbitMQ RPM包需要sudo权限(或使用root用户)才能安装和管理。"></a>2.1在基于RPM的Linux上安装（RHEL，CentOS，Fedora，openSUSE），RabbitMQ RPM包需要sudo权限(或使用root用户)才能安装和管理。</h3><figure class="highlight autoit"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">####1.安装Erlang</span></span><br><span class="line"></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># yum install erlang</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># yum update erlang</span></span><br><span class="line"></span><br><span class="line"><span class="meta">####2.安装RabbitMQ服务器</span></span><br><span class="line"> <span class="meta">##下载rabbitMQ rpm安装包</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># wget https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.7.8/rabbitmq-server-3.7.8-1.el6.noarch.rpm</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># yum install rabbitmq-server-3.7.8-1.el6.noarch.rpm</span></span><br><span class="line"> <span class="meta">## 开机启动rabbitmq-server</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># chkconfig rabbitmq-server on</span></span><br><span class="line"> <span class="meta">#启动rabbitMQ服务</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># service rabbitmq-server start</span></span><br><span class="line"> <span class="meta">#停止rabbitMQ服务</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># service rabbitmq-server stop</span></span><br><span class="line"> <span class="meta">#重启rabbitMQ服务</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># service rabbitmq-server restart</span></span><br><span class="line"> <span class="meta">#修改配置文件</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># cd /etc/rabbitmq/</span></span><br><span class="line">[root<span class="symbol">@slave1</span> rabbitmq]<span class="meta"># cp /usr/share/doc/rabbitmq-server-3.7.8/rabbitmq.config.example ./</span></span><br><span class="line">[root<span class="symbol">@slave1</span> rabbitmq]<span class="meta"># mv rabbitmq.config.example rabbitmq.config</span></span><br><span class="line">[root<span class="symbol">@slave1</span> rabbitmq]<span class="meta"># vim rabbitmq.config</span></span><br><span class="line">[</span><br><span class="line">{rabbit, [{tcp_listeners, [<span class="number">5672</span>]}, {loopback_users, [<span class="string">"test"</span>]}]}</span><br><span class="line">].</span><br><span class="line"></span><br><span class="line"> <span class="meta">#开启Web管理插件，这样我们就可以通过浏览器来进行管理了</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># rabbitmq-plugins enable rabbitmq_management</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># service rabbitmq-server restart</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># vim /etc/rabbitmq/rabbitmq.config</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># service rabbitmq-server restart</span></span><br><span class="line"> <span class="meta">##增加rabbitmq ui的登录用户</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># rabbitmqctl add_user test 123456</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># rabbitmqctl set_user_tags test administrator</span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># rabbitmqctl set_permissions -p <span class="string">"/"</span> test <span class="string">".*"</span> <span class="string">".*"</span> <span class="string">".*"</span></span></span><br><span class="line">[root<span class="symbol">@slave1</span> ~]<span class="meta"># rabbitmqctl list_users</span></span><br><span class="line">Listing users</span><br><span class="line">test[administrator]</span><br><span class="line">guest[administrator]</span><br></pre></td></tr></table></figure><h3 id="2-2-在macOS中安装rabbitMQ"><a href="#2-2-在macOS中安装rabbitMQ" class="headerlink" title="2.2 在macOS中安装rabbitMQ"></a>2.2 在macOS中安装rabbitMQ</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">使用homebrew安装rabbitMQ,若安装过homebrew可以忽略下面第一句命令</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> /usr/bin/ruby -e <span class="string">"<span class="variable">$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)</span>"</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">$</span><span class="bash"> brew update</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> brew install rabbitmq</span></span><br><span class="line"><span class="meta">#</span><span class="bash">启动命令</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> brew services start rabbitmq</span></span><br><span class="line"><span class="meta">#</span><span class="bash">重启命令</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> brew services restart rabbitmq</span></span><br><span class="line"><span class="meta">#</span><span class="bash">停止命令</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> brew services stop rabbitmq</span></span><br></pre></td></tr></table></figure><p>使用浏览器访问ip:15672(rabbitMQ的UI界面)，可使用上面设置的test用户密码进行登录</p>]]></content>
<summary type="html">
<h2 id="0-消息中间件的介绍"><a href="#0-消息中间件的介绍" class="headerlink" title="0. 消息中间件的介绍"></a>0. 消息中间件的介绍</h2><p>RabbitMQ是一个“传统”消息代理，可以实现各种消息传递协议。它是首
</summary>
<category term="消息中间件" scheme="https://blog.monbuilder.top/categories/%E6%B6%88%E6%81%AF%E4%B8%AD%E9%97%B4%E4%BB%B6/"/>
<category term="rabbitMQ 分布式" scheme="https://blog.monbuilder.top/tags/rabbitMQ-%E5%88%86%E5%B8%83%E5%BC%8F/"/>
</entry>
<entry>
<title>activiti6-tutorial00</title>
<link href="https://blog.monbuilder.top/2018/10/16/activiti6-tutorial00/"/>
<id>https://blog.monbuilder.top/2018/10/16/activiti6-tutorial00/</id>
<published>2018-10-16T03:02:23.000Z</published>
<updated>2018-12-14T06:14:20.396Z</updated>
<summary type="html">
</summary>
</entry>
<entry>
<title>webSphere性能监控(JMX)</title>
<link href="https://blog.monbuilder.top/2018/09/21/websphere-monitor/"/>
<id>https://blog.monbuilder.top/2018/09/21/websphere-monitor/</id>
<published>2018-09-21T08:12:32.000Z</published>
<updated>2018-12-14T06:14:20.399Z</updated>
<content type="html"><![CDATA[<h3 id="1-监控方式"><a href="#1-监控方式" class="headerlink" title="1.监控方式"></a>1.监控方式</h3><h4 id="1-1-部署自带perfServletApp监控项目"><a href="#1-1-部署自带perfServletApp监控项目" class="headerlink" title="1.1 部署自带perfServletApp监控项目"></a>1.1 部署自带perfServletApp监控项目</h4><p>以下步骤为支持JMX组件采集，websphere所需做的一些配置操作：<br>1>访问websphere的web console页面，例如：<a href="https://localhost:9043/ibm/console/" target="_blank" rel="noopener">https://localhost:9043/ibm/console/</a><br>2>点击左侧树形菜单：应用程序->应用程序类型->WebSphere企业应用程序；进入页面，查看有无安装了perfServletApp，若无再点击：安装 按钮<br><img src="/2018/09/21/websphere-monitor/img/perfServletApp_install.png" alt="perfServletApp_install"><br>3>在新新页面中选择：本地路径；找到并上传perfServletApp.ear部署包(此包路径在$WEBSPHERE_INSTALL_HOME/installableApps/PerfServletApp.ear)，<br>可以copy到本地在上传<br><img src="/2018/09/21/websphere-monitor/img/choose_install_file.png" alt="choose_install_file"><br>4>点击下一步，开始安装，一路点“下一步”，不用做任何修改填空，直到点击“完成”，最后保存配置<br><img src="/2018/09/21/websphere-monitor/img/install_success.png" alt="install_success"><br>5>启动perfServletApp程序<br><img src="/2018/09/21/websphere-monitor/img/start_servlet.png" alt="start_servlet"><br>6>配置perfServletApp访问的安全校验,点击：应用程序->perfServletApp,进入安全用户配置,按照下列步骤，最后确认并保存到主配置<br><img src="/2018/09/21/websphere-monitor/img/application_perm_open.png" alt="application_perm_open"></p><p><img src="/2018/09/21/websphere-monitor/img/servlet_access_perm.png" alt="servlet_access_perm"></p><p><img src="/2018/09/21/websphere-monitor/img/perm_user_mapping.png" alt="perm_user_mapping"></p><p><img src="/2018/09/21/websphere-monitor/img/perm_choose_user.png" alt="perm_choose_user"></p><p>7>需要以上配置生效，需要重启webSphere服务</p><h4 id="1-2-配置新增ssl证书-通过soap协议访问mbean"><a href="#1-2-配置新增ssl证书-通过soap协议访问mbean" class="headerlink" title="1.2 配置新增ssl证书,通过soap协议访问mbean"></a>1.2 配置新增ssl证书,通过soap协议访问mbean</h4><p>以下步骤为支持JMX组件采集，websphere所需做的一些配置操作：</p><p><img src="/2018/09/21/websphere-monitor/img/ikeyman_path.png" alt="ikeyman_path"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_1.png" alt="keystore_create_1"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_2.png" alt="keystore_create_2"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_3.png" alt="keystore_create_3"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_4.png" alt="keystore_create_4"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_5.png" alt="keystore_create_5"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_6.png" alt="keystore_create_6"><br><img src="/2018/09/21/websphere-monitor/img/keystore_create_7.png" alt="keystore_create_7"><br><img src="/2018/09/21/websphere-monitor/img/truststore_create_1.png" alt="truststore_create_1"><br><img src="/2018/09/21/websphere-monitor/img/truststore_create_2.png" alt="truststore_create_2"><br><img src="/2018/09/21/websphere-monitor/img/truststore_create_3.png" alt="truststore_create_3"><br><img src="/2018/09/21/websphere-monitor/img/truststore_create_4.png" alt="truststore_create_4"><br><img src="/2018/09/21/websphere-monitor/img/add_ssl_store_1.png" alt="add_ssl_store_1"><br><img src="/2018/09/21/websphere-monitor/img/add_ssl_store_2.png" alt="add_ssl_store_2"><br><img src="/2018/09/21/websphere-monitor/img/add_ssl_store_3.png" alt="add_ssl_store_3"><br><img src="/2018/09/21/websphere-monitor/img/add_ssl_store_4.png" alt="add_ssl_store_4"><br><img src="/2018/09/21/websphere-monitor/img/add_ssl_store_5.png" alt="add_ssl_store_5"></p><p><img src="/2018/09/21/websphere-monitor/img/stopserver_new_ssl_error.png" alt="stopserver_new_ssl_error"></p>]]></content>
<summary type="html">
<h3 id="1-监控方式"><a href="#1-监控方式" class="headerlink" title="1.监控方式"></a>1.监控方式</h3><h4 id="1-1-部署自带perfServletApp监控项目"><a href="#1-1-部署自带per
</summary>
<category term="webSphere" scheme="https://blog.monbuilder.top/categories/webSphere/"/>
<category term="性能监控" scheme="https://blog.monbuilder.top/tags/%E6%80%A7%E8%83%BD%E7%9B%91%E6%8E%A7/"/>
</entry>
<entry>
<title>搭建大数据平台系列(1)-Hadoop环境搭建[hdfs,yarn,mapreduce]</title>
<link href="https://blog.monbuilder.top/2018/08/09/hadoop/"/>
<id>https://blog.monbuilder.top/2018/08/09/hadoop/</id>
<published>2018-08-09T15:00:08.000Z</published>
<updated>2018-12-14T06:14:20.398Z</updated>
<content type="html"><![CDATA[<h2 id="1-ssh免密码登录设置"><a href="#1-ssh免密码登录设置" class="headerlink" title="1.ssh免密码登录设置"></a>1.ssh免密码登录设置</h2><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ ssh -version</span><br><span class="line">OpenSSH_5<span class="number">.3</span>p1, OpenSSL <span class="number">1.0</span><span class="number">.1</span>e-fips <span class="number">11</span> Feb <span class="number">2013</span></span><br><span class="line">Bad escape character 'rsion'.</span><br></pre></td></tr></table></figure><p>查看ssh的版本后，如果ssh未安装则需要执行如下安装命令：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> sudo yum install openssh-server</span><br></pre></td></tr></table></figure></p><p>在每台机器上都执行一次下面的命令：<br><figure class="highlight tap"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ ssh-keygen –t rsa <span class="comment">#一路回车，提示要填的都默认不填，按回车</span></span><br><span class="line">上面执行完成后，每台机器上都会生成一个~/.ssh文件夹</span><br><span class="line">$ ll ~/.ssh <span class="comment">#查看.ssh文件下的文件列表</span></span><br><span class="line">-rw-------.<span class="number"> 1 </span>hadoop hadoop<span class="number"> 1580 </span>Apr<span class="number"> 18 </span>16:53 authorized_keys</span><br><span class="line">-rw-------.<span class="number"> 1 </span>hadoop hadoop<span class="number"> 1675 </span>Apr<span class="number"> 15 </span>16:01 id_rsa</span><br><span class="line">-rw-r--r--.<span class="number"> 1 </span>hadoop hadoop <span class="number"> 395 </span>Apr<span class="number"> 15 </span>16:01 id_rsa.pub</span><br></pre></td></tr></table></figure></p><p>把slave1，slave2，slave3上生成的公钥id_rsa.pub发给master机器：<br>在slave1机器上：<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop<span class="variable">@slave1</span> ~]<span class="variable">$ </span>scp ~<span class="regexp">/.ssh/id</span>_rsa.pub hadoop<span class="variable">@master</span><span class="symbol">:~/</span>.ssh/id_rsa.pub.slave1</span><br></pre></td></tr></table></figure></p><p>在slave2机器上：<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop<span class="variable">@slave2</span> ~]<span class="variable">$ </span>scp ~<span class="regexp">/.ssh/id</span>_rsa.pub hadoop<span class="variable">@master</span><span class="symbol">:~/</span>.ssh/id_rsa.pub.slave2</span><br></pre></td></tr></table></figure></p><p>在slave3机器上：<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop<span class="variable">@slave3</span> ~]<span class="variable">$ </span>scp ~<span class="regexp">/.ssh/id</span>_rsa.pub hadoop<span class="variable">@master</span><span class="symbol">:~/</span>.ssh/id_rsa.pub.slave3</span><br></pre></td></tr></table></figure></p><p>在master机器上，将所有公钥加到新增的用于认证的公钥文件authorized_keys中：<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ cat ~<span class="regexp">/.ssh/id</span>_rsa.pub* <span class="meta">>> </span>~<span class="regexp">/.ssh/authorized</span>_keys</span><br></pre></td></tr></table></figure></p><p>需要修改文件authorized_keys的权限（权限的设置非常重要，因为不安全的设置安全设置,会让你不能使用RSA功能 ）<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> chmod <span class="number">600</span> ~/.ssh/authorized_keys <span class="comment">#如果免密码不成功有可能缺少这步</span></span><br></pre></td></tr></table></figure></p><p>将公钥文件authorized_keys分发给每台slave:<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span> scp ~<span class="regexp">/.ssh/authorized</span>_keys hadoop<span class="variable">@slave1</span><span class="symbol">:~/</span>.ssh/</span><br><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span> scp ~<span class="regexp">/.ssh/authorized</span>_keys hadoop<span class="variable">@slave1</span><span class="symbol">:~/</span>.ssh/</span><br><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span> scp ~<span class="regexp">/.ssh/authorized</span>_keys hadoop<span class="variable">@slave1</span><span class="symbol">:~/</span>.ssh/</span><br></pre></td></tr></table></figure></p><h2 id="2-Java环境的安装"><a href="#2-Java环境的安装" class="headerlink" title="2.Java环境的安装"></a>2.Java环境的安装</h2><pre><code>下载jdk-8u60-linux-x64.tar.gz安装包后(放在~/bigdataspace路径下)：</code></pre> <figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> cd ~/bigdataspace</span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ tar -zxvf jdk-<span class="number">8</span>u60-linux-x64.tar.gz</span><br></pre></td></tr></table></figure><p>修改环境变量配置文件:<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master bigdataspace]$ sudo vi /etc/profile</span><br><span class="line"></span><br><span class="line">(在配置文件末尾加上如下配置)</span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">JAVA_HOME</span>=/home/hadoop/bigdataspace/jdk1.8.0_60</span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">PATH</span>=<span class="variable">$JAVA_HOME</span>/bin:$PATH</span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">CLASSPATH</span>=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar</span><br></pre></td></tr></table></figure></p><p>让环境变量设置生效：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ source /etc/profile</span><br></pre></td></tr></table></figure></p><p>验证Java是否安装成功：<br><figure class="highlight mipsasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master <span class="keyword">bigdataspace]$ </span> <span class="keyword">java </span> -version</span><br><span class="line"><span class="keyword">java </span>version <span class="string">"1.8.0_60"</span></span><br><span class="line"><span class="keyword">Java(TM) </span>SE Runtime Environment (<span class="keyword">build </span><span class="number">1</span>.<span class="number">8</span>.<span class="number">0</span>_60-<span class="keyword">b27)</span></span><br><span class="line"><span class="keyword">Java </span>HotSpot(TM) <span class="number">64</span>-<span class="keyword">Bit </span>Server VM (<span class="keyword">build </span><span class="number">25</span>.<span class="number">60</span>-<span class="keyword">b23, </span>mixed mode)</span><br></pre></td></tr></table></figure></p><p>(每台机器上都需要按照上面的操作安装Java)<br>每台机器上执行：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> sudo chmod <span class="number">777</span> /data/ <span class="comment">#让所有用户可操作/data目录下的数据</span></span><br></pre></td></tr></table></figure></p><h2 id="3-集群上的机器实现同步时间"><a href="#3-集群上的机器实现同步时间" class="headerlink" title="3.集群上的机器实现同步时间"></a>3.集群上的机器实现同步时间</h2><p>检查时间服务是否安装:<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ rpm -q ntp</span><br><span class="line">ntp-4.2.6p5-1.el6.centos.x86_64 #这表示已安装了，如果没有安装，这是空白</span><br></pre></td></tr></table></figure></p><p>如果没有安装，需要执行下面的安装命令：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> sudo yum install ntp</span><br></pre></td></tr></table></figure></p><p>需要配置NTP服务为自启动:<br><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span>sudo chkconfig ntpd on</span><br><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span>chkconfig --list ntpd</span><br><span class="line">ntpd <span class="number">0</span><span class="symbol">:off</span> <span class="number">1</span><span class="symbol">:off</span> <span class="number">2</span><span class="symbol">:on</span> <span class="number">3</span><span class="symbol">:on</span> <span class="number">4</span><span class="symbol">:on</span> <span class="number">5</span><span class="symbol">:on</span> <span class="number">6</span><span class="symbol">:off</span></span><br><span class="line"></span><br><span class="line">(需要打开master机器上udp协议的<span class="number">123</span>端口是为了其他节点使用ntpdate通过该端口同步master机器的时间)</span><br><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span>sudo vi /etc/sysconfig/iptables</span><br><span class="line">(新增的端口配置)</span><br><span class="line">-A INPUT -m state --state NEW -m udp -p udp --dport <span class="number">123</span> -j ACCEPT</span><br><span class="line">[hadoop<span class="variable">@master</span> ~]<span class="variable">$ </span>sudo service iptables restart</span><br></pre></td></tr></table></figure></p><p>在配置前，先使用ntpdate手动同步下时间，免得本机与外部时间服务器时间差距太大，让ntpd不能正常同步。<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ sudo ntpdate pool.ntp.org</span><br><span class="line">26 Apr 17:12:15 ntpdate[7376]: <span class="keyword">step</span> time<span class="built_in"> server </span>202.112.29.82 offset 13.827386 sec</span><br></pre></td></tr></table></figure></p><p>更改master机器上的相关配置文件:<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> sudo vim /etc/ntp.conf</span><br></pre></td></tr></table></figure></p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line">(下面只显示修改的必要项)</span><br><span class="line"><span class="comment"># Hosts on local network are less restricted.</span></span><br><span class="line">restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap</span><br><span class="line"><span class="comment">#让同一局域网ip段可以进行时间同步：</span></span><br><span class="line">restrict 10.3.19.0 mask 255.255.255.0 nomodify notrap</span><br><span class="line"><span class="comment"># Use public servers from the pool.ntp.org project.</span></span><br><span class="line"><span class="comment"># Please consider joining the pool (http://www.pool.ntp.org/join.html).</span></span><br><span class="line"><span class="comment">#server 0.centos.pool.ntp.org iburst</span></span><br><span class="line"><span class="comment">#server 1.centos.pool.ntp.org iburst</span></span><br><span class="line"><span class="comment">#server 2.centos.pool.ntp.org iburst</span></span><br><span class="line"><span class="comment">#server 3.centos.pool.ntp.org iburst</span></span><br><span class="line"><span class="comment">#外部时间服务器</span></span><br><span class="line">server pool.ntp.org iburst</span><br><span class="line">server 0.asia.pool.ntp.org iburst</span><br><span class="line">server 1.asia.pool.ntp.org iburst</span><br><span class="line">server 1.asia.pool.ntp.org iburst</span><br><span class="line">server 2.asia.pool.ntp.org iburst</span><br><span class="line"><span class="comment">#broadcast 192.168.1.255 autokey # broadcast server</span></span><br><span class="line"><span class="comment">#broadcastclient # broadcast client</span></span><br><span class="line"><span class="comment">#broadcast 224.0.1.1 autokey # multicast server</span></span><br><span class="line"><span class="comment">#multicastclient 224.0.1.1 # multicast client</span></span><br><span class="line"><span class="comment">#manycastserver 239.255.254.254 # manycast server</span></span><br><span class="line"><span class="comment">#manycastclient 239.255.254.254 autokey # manycast client</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># allow update time by the upper server</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Undisciplined Local Clock. This is a fake driver intended for backup</span></span><br><span class="line"><span class="comment"># and when no outside source of synchronized time is available.</span></span><br><span class="line"><span class="comment"># 外部时间服务器不可用时，以本地时间作为时间服务</span></span><br><span class="line">server 127.127.1.0</span><br><span class="line">fudge 127.127.1.0 stratum 10</span><br><span class="line"></span><br><span class="line"><span class="comment">#############################################################</span></span><br><span class="line">其他节点/etc/ntp.conf（slave1,slave2,slave3）的配置：</span><br><span class="line">……<span class="built_in">..</span></span><br><span class="line"><span class="comment">#server 3.centos.pool.ntp.org iburst</span></span><br><span class="line"><span class="comment">#外部时间服务器，以master时间为准进行同步</span></span><br><span class="line">server master iburst</span><br><span class="line">……<span class="built_in">..</span></span><br></pre></td></tr></table></figure><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ sudo <span class="built_in"> service </span> ntpd start</span><br><span class="line">(每台机器上都需要，设置ntpd开机启动，并第一次手动打开ntpd)，命令如下：</span><br><span class="line">$ sudo chkconfig ntpd on #开机启动ntpd</span><br><span class="line">$ sudo<span class="built_in"> service </span>ntpd start #启动 ntpd</span><br></pre></td></tr></table></figure><p>时间同步设置参考：<a href="http://cn.soulmachine.me/blog/20140124/" target="_blank" rel="noopener">http://cn.soulmachine.me/blog/20140124/</a></p><p>时间同步设置总结：<br> 每个节点上安装ntpd，并设置为开机启动，当然第一次要先手动启动，通过配置/etc/ntp.conf文件，让master作为时间同步服务器，这台机器的时间是根据联网同步网络时间的，其他节点以master的ip作为同步的地址</p><p>配置完成后，发现后面的节点时间可能还未同步，可能需要等30分钟左右，一段时间后时间都会以master为准，进行同步</p><h2 id="4-Hadoop的安装、配置"><a href="#4-Hadoop的安装、配置" class="headerlink" title="4.Hadoop的安装、配置"></a>4.Hadoop的安装、配置</h2><p>下载hadoop-2.6.0-cdh5.5.0.tar.gz安装包后(放在master机器上的~/bigdataspace路径下)：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> cd ~/bigdataspace</span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ tar -zxvf hadoop-<span class="number">2.6</span>.<span class="number">0</span>-cdh5.<span class="number">5.0</span>.tar.gz</span><br></pre></td></tr></table></figure></p><p>进入hadoop配置文件路径:<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ cd ~/bigdataspace/hadoop<span class="number">-2.6</span><span class="number">.0</span>-cdh5<span class="number">.5</span><span class="number">.0</span>/etc/hadoop</span><br></pre></td></tr></table></figure></p><h3 id="1-gt-在hadoop-env-sh中配置JAVA-HOME："><a href="#1-gt-在hadoop-env-sh中配置JAVA-HOME：" class="headerlink" title="1> 在hadoop-env.sh中配置JAVA_HOME："></a>1> 在hadoop-env.sh中配置JAVA_HOME：</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi hadoop-env.sh</span><br></pre></td></tr></table></figure><figure class="highlight delphi"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"># <span class="keyword">set</span> JAVA_HOME <span class="keyword">in</span> this <span class="keyword">file</span>, so that it <span class="keyword">is</span> correctly defined <span class="keyword">on</span></span><br><span class="line"># The java <span class="keyword">implementation</span> <span class="keyword">to</span> use.</span><br><span class="line"><span class="keyword">export</span> JAVA_HOME=/home/hadoop/bigdataspace/jdk1.<span class="number">8.0</span>_60</span><br></pre></td></tr></table></figure><h3 id="2-gt-在yarn-env-sh中配置JAVA-HOME："><a href="#2-gt-在yarn-env-sh中配置JAVA-HOME：" class="headerlink" title="2> 在yarn-env.sh中配置JAVA_HOME："></a>2> 在yarn-env.sh中配置JAVA_HOME：</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi yarn-env.sh</span><br></pre></td></tr></table></figure><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># some Java parameters</span></span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">JAVA_HOME</span>=/home/hadoop/bigdataspace/jdk1.8.0_60</span><br></pre></td></tr></table></figure><h3 id="3-gt-在slaves中配置slave节点的ip或者host"><a href="#3-gt-在slaves中配置slave节点的ip或者host" class="headerlink" title="3> 在slaves中配置slave节点的ip或者host"></a>3> 在slaves中配置slave节点的ip或者host</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi slaves</span><br></pre></td></tr></table></figure><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">slave1</span><br><span class="line">slave2</span><br><span class="line">slave3</span><br></pre></td></tr></table></figure><h3 id="4-gt-修改core-site-xml"><a href="#4-gt-修改core-site-xml" class="headerlink" title="4> 修改core-site.xml"></a>4> 修改core-site.xml</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi core-site.xml</span><br></pre></td></tr></table></figure><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"><!-- Put site-specific property overrides in this file. --></span></span><br><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>fs.defaultFS<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hdfs://master:8020<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.tmp.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/hadoop-2.6.0-cdh5.5.0/tmp<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><h3 id="5-gt-修改hdfs-site-xml"><a href="#5-gt-修改hdfs-site-xml" class="headerlink" title="5> 修改hdfs-site.xml"></a>5> 修改hdfs-site.xml</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi hdfs-site.xml</span><br></pre></td></tr></table></figure><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"><!-- Put site-specific property overrides in this file. --></span></span><br><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.secondary.http-address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:50090<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.name.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>file:/data/hadoop-2.6.0-cdh5.5.0/dfs/name<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.data.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"><span class="tag"><<span class="name">name</span>></span>dfs.datanode.data.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>file:/data/hadoop-2.6.0-cdh5.5.0/dfs/data<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.replication<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>3<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><h3 id="6-gt-修改mapred-site-xml"><a href="#6-gt-修改mapred-site-xml" class="headerlink" title="6> 修改mapred-site.xml"></a>6> 修改mapred-site.xml</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi mapred-site.xml</span><br></pre></td></tr></table></figure><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"><!-- Put site-specific property overrides in this file. --></span></span><br><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>mapreduce.framework.name<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>yarn<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>mapreduce.jobhistory.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:10020<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>mapreduce.jobhistory.webapp.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:19888<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><h3 id="7-gt-修改yarn-site-xml"><a href="#7-gt-修改yarn-site-xml" class="headerlink" title="7> 修改yarn-site.xml"></a>7> 修改yarn-site.xml</h3><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop</span>]$ vi yarn-site.xml</span><br></pre></td></tr></table></figure><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"><span class="comment"><!-- Site specific YARN configuration properties --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.nodemanager.aux-services<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>mapreduce_shuffle<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.nodemanager.aux-services.mapreduce.shuffle.class<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>org.apache.hadoop.mapred.ShuffleHandler<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.resourcemanager.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:8032<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.resourcemanager.scheduler.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:8030<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.resourcemanager.resource-tracker.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:8031<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.resourcemanager.admin.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:8033<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.resourcemanager.webapp.address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:8088<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure><p><em>因为CDH版本缺少hadoop的native库，因此需要引入，否则会报错，解决方法：</em><br><a href="http://www.cnblogs.com/huaxiaoyao/p/5046374.html" target="_blank" rel="noopener">http://www.cnblogs.com/huaxiaoyao/p/5046374.html</a><br>本次安装具体采取的解决方法：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> cd ~/bigdataspace</span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ wget http://archive.cloudera.com/cdh5/redhat/<span class="number">6</span>/x86_64/cdh/<span class="number">5.5</span>.<span class="number">0</span>/RPMS/x86_64/hadoop-<span class="number">2.6</span>.<span class="number">0</span>+cdh5.<span class="number">5.0</span>+<span class="number">921</span>-<span class="number">1</span>.cdh5.<span class="number">5.0</span>.p0.<span class="number">15</span>.el6.x86_64.rpm</span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ rpm2cpio *.rpm | cpio -div</span><br></pre></td></tr></table></figure></p><p>在bigdataspace文件夹下<br><figure class="highlight crystal"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ cp -r ./usr/<span class="class"><span class="keyword">lib</span>/<span class="title">hadoop</span>/<span class="title">lib</span>/<span class="title">native</span>/ ~/<span class="title">bigdataspace</span>/<span class="title">hadoop</span>-2.6.0-<span class="title">cdh5</span>.5.0/<span class="title">lib</span>/<span class="title">native</span>/</span></span><br></pre></td></tr></table></figure></p><p>删除解压后得到的文件：<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master bigdataspace]$ rm -r ~/bigdataspace/etc/</span><br><span class="line">[hadoop@master bigdataspace]$ rm -r ~/bigdataspace/usr/</span><br><span class="line">[hadoop@master bigdataspace]$ rm -r ~/bigdataspace/var<span class="comment">//</span></span><br><span class="line">$ rm ~/ bigdataspace/hadoop<span class="number">-2.6</span><span class="number">.0</span>+cdh5<span class="number">.5</span><span class="number">.0</span>+<span class="number">921</span><span class="number">-1.</span>cdh5<span class="number">.5</span><span class="number">.0</span>.p0<span class="number">.15</span>.el6.x86_64.rpm</span><br></pre></td></tr></table></figure></p><h2 id="5-使用scp命令分发配置好的hadoop到各个子节点"><a href="#5-使用scp命令分发配置好的hadoop到各个子节点" class="headerlink" title="5.使用scp命令分发配置好的hadoop到各个子节点"></a>5.使用scp命令分发配置好的hadoop到各个子节点</h2><figure class="highlight elixir"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="variable">$ </span> scp –r ~<span class="regexp">/bigdataspace/hadoop</span>-<span class="number">2.6</span>.<span class="number">0</span>-cdh5.<span class="number">5.0</span>/ hadoop<span class="variable">@slave1</span><span class="symbol">:~/bigdataspace/</span></span><br><span class="line"><span class="variable">$ </span> scp –r ~<span class="regexp">/bigdataspace/hadoop</span>-<span class="number">2.6</span>.<span class="number">0</span>-cdh5.<span class="number">5.0</span>/ hadoop<span class="variable">@slave2</span><span class="symbol">:~/bigdataspace/</span></span><br><span class="line"><span class="variable">$ </span> scp –r ~<span class="regexp">/bigdataspace/hadoop</span>-<span class="number">2.6</span>.<span class="number">0</span>-cdh5.<span class="number">5.0</span>/ hadoop<span class="variable">@slave3</span><span class="symbol">:~/bigdataspace/</span></span><br></pre></td></tr></table></figure><p>(每台机器)修改环境变量配置文件:<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ sudo vi /etc/profile</span><br></pre></td></tr></table></figure></p><p>(在配置文件末尾加上如下配置)<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="builtin-name">export</span> <span class="attribute">HADOOP_HOME</span>=/home/hadoop/bigdataspace/hadoop-2.6.0-cdh5.5.0</span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">PATH</span>=<span class="variable">$JAVA_HOME</span>/bin:$HADOOP_HOME/bin:$PATH</span><br></pre></td></tr></table></figure></p><p>让环境变量设置生效：<br><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">bigdataspace</span>]$ source /etc/profile</span><br></pre></td></tr></table></figure></p><h2 id="6-启动并验证Hadoop"><a href="#6-启动并验证Hadoop" class="headerlink" title="6.启动并验证Hadoop"></a>6.启动并验证Hadoop</h2><figure class="highlight crmsh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">~]$</span> cd ~/bigdataspace/hadoop-<span class="number">2.6</span>.<span class="number">0</span>-cdh5.<span class="number">5.0</span> <span class="comment">#进入hadoop目录</span></span><br><span class="line"> [hadoop@<span class="keyword">master</span> <span class="title">hadoop-2</span>.<span class="number">6.0</span>-cdh5.<span class="number">5.0</span>]$ ./bin/hdfs namenode –format <span class="comment">#格式化namenode</span></span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop-2</span>.<span class="number">6.0</span>-cdh5.<span class="number">5.0</span>]$ ./sbin/<span class="literal">start</span>-dfs.sh <span class="comment">#启动dfs</span></span><br><span class="line">[hadoop@<span class="keyword">master</span> <span class="title">hadoop-2</span>.<span class="number">6.0</span>-cdh5.<span class="number">5.0</span>]$ ./sbin/<span class="literal">start</span>-yarn.sh <span class="comment">#启动yarn</span></span><br></pre></td></tr></table></figure><p>可以通过jps命令查看各个节点启动的进程是否正常。在 master 上应该有以下几个进程<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master hadoop<span class="number">-2.6</span><span class="number">.0</span>-cdh5<span class="number">.5</span><span class="number">.0</span>]$ jps</span><br><span class="line"><span class="number">3407</span> SecondaryNameNode</span><br><span class="line"><span class="number">3218</span> NameNode</span><br><span class="line"><span class="number">3552</span> ResourceManager</span><br><span class="line"><span class="number">3910</span> Jps</span><br></pre></td></tr></table></figure></p><p>在 slave1 上应该有以下几个进程<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@slave1 ~]$ jps</span><br><span class="line"><span class="number">2072</span> NodeManager</span><br><span class="line"><span class="number">2213</span> Jps</span><br><span class="line"><span class="number">1962</span> DataNode</span><br></pre></td></tr></table></figure></p><p><em>或者在浏览器中输入 <a href="http://master:8088" target="_blank" rel="noopener">http://master:8088</a> ，应该有 hadoop 的管理界面出来了,并通过<a href="http://master:8088/cluster/nodes能看到" target="_blank" rel="noopener">http://master:8088/cluster/nodes能看到</a> slave1、slave2、slave3节点</em></p><h2 id="7-启动Hadoop自带的jobhistoryserver"><a href="#7-启动Hadoop自带的jobhistoryserver" class="headerlink" title="7.启动Hadoop自带的jobhistoryserver"></a>7.启动Hadoop自带的jobhistoryserver</h2><p>[hadoop@master ~]$ cd ~/bigdataspace/hadoop-2.6.0-cdh5.5.0 #进入hadoop目录<br>[hadoop@master hadoop-2.6.0-cdh5.5.0]$ sbin/mr-jobhistory-daemon.sh start historyserver<br>(mapred-site.xml配置文件有对jobhistory的相关配置)<br>[hadoop@master hadoop-2.6.0-cdh5.5.0]$ jps<br>5314 Jps<br>19994 JobHistoryServer<br>19068 NameNode<br>19422 ResourceManager<br>19263 SecondaryNameNode</p><p>参考：<br><a href="http://blog.csdn.net/liubei_whut/article/details/42397985" target="_blank" rel="noopener">http://blog.csdn.net/liubei_whut/article/details/42397985</a></p><h2 id="8-停止hadoop集群的问题"><a href="#8-停止hadoop集群的问题" class="headerlink" title="8.停止hadoop集群的问题"></a>8.停止hadoop集群的问题</h2><p>Linux运行一段时间后，/tmp下的文件夹下面会清空一些文件，hadoop的停止脚本stop-all.sh是需要根据/tmp下面的pid文件关闭对应的进程，当/tmp下的文件被自动清理后可能会出出先的错误：<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ ./sbin/<span class="keyword">stop</span>-all.sh</span><br><span class="line">Stopping namenodes <span class="keyword">on</span> [<span class="keyword">master</span>]</span><br><span class="line"><span class="keyword">master</span>: <span class="keyword">no</span> namenode <span class="keyword">to</span> <span class="keyword">stop</span></span><br><span class="line">slave1: <span class="keyword">no</span> datanode <span class="keyword">to</span> <span class="keyword">stop</span></span><br><span class="line">slave2: <span class="keyword">no</span> datanode <span class="keyword">to</span> <span class="keyword">stop</span></span><br><span class="line">slave3: <span class="keyword">no</span> datanode <span class="keyword">to</span> <span class="keyword">stop</span></span><br><span class="line">Stopping secondary namenodes [<span class="keyword">master</span>]</span><br><span class="line"><span class="keyword">master</span>: <span class="keyword">no</span> secondarynamenode <span class="keyword">to</span> <span class="keyword">stop</span></span><br><span class="line">……</span><br></pre></td></tr></table></figure></p><p><em>方法1：这时需要在/tmp文件夹下手动创建恢复这些pid文件</em><br>master节点（每个文件中保存对应的进程id）：<br>hadoop-hadoop-namenode.pid<br>hadoop-hadoop-secondarynamenode.pid<br>yarn-hadoop-resourcemanager.pid<br>slave节点（每个文件中保存对应的进程id）：<br>hadoop-hadoop-datanode.pid<br>yarn-hadoop-nodemanager.pid<br><em>方法2：使用kill -9逐个关闭相应的进程id</em></p><p>从根本上解决的方法：<br>（首先使用了方法1或方法2关闭了hadoop集群）<br>1.修改配置文件hadoop-env.sh:<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#export HADOOP_PID_DIR=${HADOOP_PID_DIR}</span></span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">HADOOP_PID_DIR</span>=/data/hadoop-2.6.0-cdh5.5.0/pids</span><br><span class="line"><span class="comment">#export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}</span></span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">HADOOP_SECURE_DN_PID_DIR</span>=/data/hadoop-2.6.0-cdh5.5.0/pids</span><br></pre></td></tr></table></figure></p><p>2.修改配置文件yarn-env.sh:<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">export YARN_PID_DIR=/data/hadoop<span class="number">-2.6</span><span class="number">.0</span>-cdh5<span class="number">.5</span><span class="number">.0</span>/pids</span><br></pre></td></tr></table></figure></p><p>3.创建文件夹pids：<br><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir /data/hadoop<span class="number">-2.6</span><span class="number">.0</span>-cdh5<span class="number">.5</span><span class="number">.0</span>/pids（发现会自动创建pids文件，因此不需要创建）</span><br></pre></td></tr></table></figure></p><p>这2个步骤需要在各个节点都执行.</p>]]></content>
<summary type="html">
<h2 id="1-ssh免密码登录设置"><a href="#1-ssh免密码登录设置" class="headerlink" title="1.ssh免密码登录设置"></a>1.ssh免密码登录设置</h2><figure class="highlight lsl"><ta
</summary>
<category term="大数据" scheme="https://blog.monbuilder.top/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Hadoop" scheme="https://blog.monbuilder.top/tags/Hadoop/"/>
</entry>
<entry>
<title>搭建大数据平台系列(0)-机器准备</title>
<link href="https://blog.monbuilder.top/2018/07/20/conf-os/"/>
<id>https://blog.monbuilder.top/2018/07/20/conf-os/</id>
<published>2018-07-20T05:16:05.000Z</published>
<updated>2018-12-14T06:14:20.376Z</updated>
<content type="html"><![CDATA[<h2 id="0-前期规划"><a href="#0-前期规划" class="headerlink" title="0. 前期规划"></a>0. 前期规划</h2><p>假设现在有四台机器，各自ip如下:<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">192<span class="selector-class">.168</span><span class="selector-class">.1</span><span class="selector-class">.201</span></span><br><span class="line">192<span class="selector-class">.168</span><span class="selector-class">.1</span><span class="selector-class">.202</span></span><br><span class="line">192<span class="selector-class">.168</span><span class="selector-class">.1</span><span class="selector-class">.203</span></span><br><span class="line">192<span class="selector-class">.168</span><span class="selector-class">.1</span><span class="selector-class">.204</span></span><br></pre></td></tr></table></figure></p><p>安装下面统一要求进行重装系统：</p><ol><li>每台机器的系统为：CentOS-6.5-x86_64</li><li><p>每台机器的磁盘分区为：</p><figure class="highlight ada"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">/boot : 系统引导分区</span><br><span class="line">/ : 系统安装区</span><br><span class="line">/data : 系统数据分区</span><br><span class="line">（swap分区尚未建立，待机器内存不足需求时再建立）</span><br></pre></td></tr></table></figure></li><li><p>每台机器上安装的都是Mininal Desktop版本</p></li><li>每台机器的主机名(hostname)分别为：master、slave1、slave2、slave3</li><li>每台机器的root密码都是:master</li><li>每台机器上建立的第一个非root用户都为:hadoop，密码为:hadoop</li><li>非root用户在centos中获取sudo权限的命令：<figure class="highlight fortran"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@slave3 ~]$ su - #切换到root用户，需要输入root密码</span><br><span class="line">[root@slave3 ~]# visudo -f /etc/sudoers</span><br><span class="line">## Allow root to run <span class="built_in">any</span> commands anywhere</span><br><span class="line">root <span class="built_in">ALL</span>=(<span class="built_in">ALL</span>) <span class="built_in">ALL</span></span><br><span class="line">在上面这条文字下加上如下：</span><br><span class="line">hadoop <span class="built_in">ALL</span>=(<span class="built_in">ALL</span>) <span class="built_in">ALL</span></span><br></pre></td></tr></table></figure></li></ol><p>这样配置后，hadoop用户可以使用sudo命令了</p><ol start="8"><li>修改每台机器的/etc/hosts文件，如：<figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@slave3 ~]$ sudo vi /etc/hosts</span><br><span class="line"><span class="number">192.168</span><span class="number">.1</span><span class="number">.201</span> master</span><br><span class="line"><span class="number">192.168</span><span class="number">.1</span><span class="number">.202</span> slave1</span><br><span class="line"><span class="number">192.168</span><span class="number">.1</span><span class="number">.203</span> slave2</span><br><span class="line"><span class="number">192.168</span><span class="number">.1</span><span class="number">.204</span> slave3</span><br></pre></td></tr></table></figure></li></ol><p>9.防火墙设置</p><ul><li>因为4台机器组成的大数据平台小集群需要互相访问对方的多个端口，需要在防火墙中打开这些端口的访问（如果未打开的话），当然最方便的就是把每台机器上的防火墙关闭了。<figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@master ~]$ sudo <span class="built_in"> service </span> iptables start #打开防火墙</span><br><span class="line">[hadoop@master ~]$ sudo <span class="built_in"> service </span> iptables status #查看防火墙</span><br><span class="line">[hadoop@master ~]$ sudo <span class="built_in"> service </span> iptables stop #关闭防火墙</span><br><span class="line">(注意：防火墙操作时需要root权限的，非root用户不使用sudo的话，不会报错，但操作无结果)</span><br><span class="line">修改防火墙设置(红色为新增配置：开放2000-6000,7000以上的端口):</span><br><span class="line">[hadoop@master ~]$ sudo vim /etc/sysconfig/iptables</span><br><span class="line"><span class="comment"># Firewall configuration written by system-config-firewall</span></span><br><span class="line"><span class="comment"># Manual customization of this file is not recommended.</span></span><br><span class="line"><span class="number">*f</span>ilter</span><br><span class="line">:INPUT ACCEPT [0:0]</span><br><span class="line">:FORWARD ACCEPT [0:0]</span><br><span class="line">:OUTPUT ACCEPT [0:0]</span><br><span class="line">-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT</span><br><span class="line">-A INPUT -p icmp -j ACCEPT</span><br><span class="line">-A INPUT -i lo -j ACCEPT</span><br><span class="line">-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT</span><br><span class="line">-A INPUT -m state --state NEW -m tcp -p tcp --dport 2000:6000 -j ACCEPT</span><br><span class="line">-A INPUT -m state --state NEW -m tcp -p tcp --dport 7000: -j ACCEPT</span><br><span class="line">-A INPUT -j REJECT --reject-with icmp-host-prohibited</span><br><span class="line">-A FORWARD -j REJECT --reject-with icmp-host-prohibited</span><br><span class="line">COMMIT</span><br><span class="line"></span><br><span class="line">保存上面的文件后，需要重启防火墙，让配置生效！</span><br><span class="line">[hadoop@master ~]$ sudo <span class="built_in"> service </span> iptables restart #重启防火墙</span><br></pre></td></tr></table></figure></li></ul>]]></content>
<summary type="html">
<h2 id="0-前期规划"><a href="#0-前期规划" class="headerlink" title="0. 前期规划"></a>0. 前期规划</h2><p>假设现在有四台机器，各自ip如下:<br><figure class="highlight css"><
</summary>
<category term="大数据" scheme="https://blog.monbuilder.top/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Hadoop" scheme="https://blog.monbuilder.top/tags/Hadoop/"/>
</entry>
<entry>
<title>Hexo+Github Page搭建个人博客</title>
<link href="https://blog.monbuilder.top/2018/06/22/build-blog/"/>
<id>https://blog.monbuilder.top/2018/06/22/build-blog/</id>
<published>2018-06-22T09:03:09.000Z</published>
<updated>2018-12-14T06:14:20.390Z</updated>
<content type="html"><![CDATA[<p>什么是Hexo？Hexo 是一个快速、简洁且高效的博客框架。Hexo 使用 Markdown（或其他渲染引擎）解析文章，在几秒内，即可利用靓丽的主题生成静态网页。堪称在座各位喜欢Markdown的优雅人士博客建站神器哟！</p><h2 id="1-Quick-Start"><a href="#1-Quick-Start" class="headerlink" title="1. Quick Start"></a>1. Quick Start</h2><h3 id="1-1-创建存放Github-Pages的仓库"><a href="#1-1-创建存放Github-Pages的仓库" class="headerlink" title="1.1 创建存放Github Pages的仓库"></a>1.1 创建存放Github Pages的仓库</h3><p>Github Pages 是面向用户、组织和项目开放的公共静态页面搭建托管服务，站点可以被免费托管在 Github 上，你可以选择使用 Github Pages 提供的域名 github.io 或者自定义域名来发布站点。</p><p>需要Github账号，请登录<a href="https://github.com/" target="_blank" rel="noopener">https://github.com/</a> 注册。<br>登录了自己的github账号后，可以安装下图一样，创建自己的GitHub Pages仓库名[参考<a href="https://pages.github.com/" target="_blank" rel="noopener">https://pages.github.com/</a> ]，<code>[PS] 仓库名repository name需要约定为: 你的账号名.github.io</code><br><img src="/2018/06/22/build-blog/img/create_pages_project.png" alt="create_page"><br>创建好博客项目仓库后，可以通过git命名下载到本地，并编辑一下README.md从本地提交到GitHub，这样做主要是使本地文件与Github关联起来，方便后面hexo deploy,直接部署博客内容到GitHub进行更新。<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> git <span class="built_in">clone</span> https://github.com/yourGithubName/yourGithubName.github.io</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> vim README.md</span></span><br><span class="line"><span class="meta">#</span><span class="bash"> REAMME.md上可以简单写一些博客介绍啥的</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> git config --global user.email <span class="string">"you@example.com"</span></span></span><br><span class="line"><span class="meta">$</span><span class="bash"> git config --global user.name <span class="string">"Your Name"</span></span></span><br><span class="line"><span class="meta">$</span><span class="bash"> git add ./</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> git commit -m <span class="string">'test'</span></span></span><br><span class="line"><span class="meta">$</span><span class="bash"> git push -u origin master</span></span><br><span class="line">Username for 'https://github.com': Builder34</span><br><span class="line">Password for 'https://Builder34@github.com':</span><br></pre></td></tr></table></figure></p><h3 id="1-2-Hexo安装"><a href="#1-2-Hexo安装" class="headerlink" title="1.2 Hexo安装"></a>1.2 Hexo安装</h3><p>安装 Hexo 相当简单。然而在安装前，您必须检查电脑中是否已安装下列应用程序：</p><ul><li>Node.js (请看<a href="https://nodejs.org/zh-cn/" target="_blank" rel="noopener">https://nodejs.org/zh-cn/</a>)</li><li>Git （请看<a href="https://git-scm.com/downloads）" target="_blank" rel="noopener">https://git-scm.com/downloads）</a><br>安装好上面2个程序后，可以进行hexo的安装了：<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> npm install -g hexo-cli</span></span><br></pre></td></tr></table></figure></li></ul><h3 id="1-3-Hexo初始化"><a href="#1-3-Hexo初始化" class="headerlink" title="1.3 Hexo初始化"></a>1.3 Hexo初始化</h3><p>安装 Hexo 完成后，我们可以在本地新建一个文件夹如：builder34.github.io(这个目录是我们Github Pages博客项目的目录),假如我的文件夹路径为/home/test/builder34.github.io，建站初始化命令可以如下:<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> <span class="built_in">cd</span> /home/<span class="built_in">test</span>/builder34.github.io</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo init ./</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> npm install</span></span><br><span class="line"></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo generate</span></span><br></pre></td></tr></table></figure></p><p>下面介绍几个常用的hexo命令(括号里面的命令为缩写形式，效果一样)：</p><pre><code>1. hexo generate(hexo g) #生成静态文件，会在当前目录下生成一个新的叫做public的文件夹2. hexo new "postTitle" #新建博客文章3. hexo new page "pageTitle" #新建1个页面4. hexo server(hexo s) #启动本地web服务预览(加参数--debug,用于调试，如：hexo s --debug)5. hexo deploy(hexo d) #部署播客到远端（比如Github,coding,heroku等平台）</code></pre><p>在命令行中输入<code>hexo s --debug</code>后，运行成功后，可以在浏览器中输入：<a href="http://localhost:4000看到自己新建的博客了。" target="_blank" rel="noopener">http://localhost:4000看到自己新建的博客了。</a></p><h3 id="1-4-更改主题"><a href="#1-4-更改主题" class="headerlink" title="1.4 更改主题"></a>1.4 更改主题</h3><p>一般我们初始化博客的文件夹后，文件结构大概如下：<br><figure class="highlight tap"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">$ ll</span><br><span class="line">total 1352</span><br><span class="line">-rw-r--r-- <span class="number"> 1 </span>builder34 staff 32B <span class="number"> 4 </span>14 01:34 README.md</span><br><span class="line">-rw-r--r-- <span class="number"> 1 </span>builder34 staff 2.3K <span class="number"> 6 </span>25 10:40 _config.yml</span><br><span class="line">-rw-r--r-- <span class="number"> 1 </span>builder34 staff 32K <span class="number"> 6 </span>26 15:50 db.json</span><br><span class="line">-rw-r--r-- <span class="number"> 1 </span>builder34 staff 458K <span class="number"> 6 </span>26 15:56 debug.log</span><br><span class="line">drwxr-xr-x <span class="number"> 293 </span>builder34 staff 9.2K <span class="number"> 6 </span>25 10:42 node_modules</span><br><span class="line">-rw-r--r-- <span class="number"> 1 </span>builder34 staff 110K <span class="number"> 6 </span>22 23:59 package-lock.json</span><br><span class="line">-rw-r--r-- <span class="number"> 1 </span>builder34 staff 564B <span class="number"> 6 </span>22 23:59 package.json</span><br><span class="line">drwxr-xr-x <span class="number"> 14 </span>builder34 staff 448B <span class="number"> 6 </span>25 10:40 public</span><br><span class="line">drwxr-xr-x <span class="number"> 5 </span>builder34 staff 160B <span class="number"> 4 </span>17 23:12 scaffolds</span><br><span class="line">drwxr-xr-x <span class="number"> 3 </span>builder34 staff 96B <span class="number"> 6 </span>25 10:57 source</span><br><span class="line">drwxr-xr-x <span class="number"> 6 </span>builder34 staff 192B <span class="number"> 6 </span>25 11:33 themes</span><br></pre></td></tr></table></figure></p><p>themes文件夹是我们博客主题的存放地方，下面我推荐一个主题：BlueLake<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> <span class="built_in">cd</span> themes/</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> git <span class="built_in">clone</span> https://github.com/chaooo/hexo-theme-BlueLake.git ./BlueLake</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> npm install hexo-renderer-jade@0.3.0 --save</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> npm install hexo-renderer-stylus --save</span></span><br><span class="line"></span><br><span class="line">(该主题更细致的配置，请登录上面这个github网址，阅读README.md进行定制化配置）</span><br></pre></td></tr></table></figure></p><p>在Hexo配置文件（$your_blog_path/_config.yml）中把主题设置修改为BlueLake。<br><figure class="highlight avrasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="symbol">theme:</span> BlueLake</span><br></pre></td></tr></table></figure></p><p>完成配置后，执行下面语句，重新打开<a href="http://localhost:4000" target="_blank" rel="noopener">http://localhost:4000</a> 可以看到博客以一个新的主题展现了<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> hexo g</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo s --debug</span></span><br></pre></td></tr></table></figure></p><h3 id="1-5-hexo部署到Github"><a href="#1-5-hexo部署到Github" class="headerlink" title="1.5 hexo部署到Github"></a>1.5 hexo部署到Github</h3><p>配置$your_blog_path/_config.yml文件的Deployment：<br><figure class="highlight dts"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta"># Deployment</span></span><br><span class="line"><span class="meta">## Docs: https:<span class="comment">//hexo.io/docs/deployment.html</span></span></span><br><span class="line"><span class="symbol">deploy:</span></span><br><span class="line"><span class="symbol"> type:</span> git</span><br><span class="line"><span class="symbol"> repo:</span> https:<span class="comment">//github.com/your_githubName/your_githubName.github.io.git</span></span><br></pre></td></tr></table></figure></p><p>通过下面的命名进行博客静态页面的生成，以及部署到远端Github Pages<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">删除静态文件,即 public 文件</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo clean</span></span><br><span class="line"><span class="meta">#</span><span class="bash">生成静态文件,即 public 文件</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo generate</span></span><br><span class="line"><span class="meta">#</span><span class="bash">部署到远程站点</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo deploy</span></span><br><span class="line"><span class="meta">#</span><span class="bash">也可以使用组合命令(替代上面2条命令)：生成静态命令并部署到远程站点</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo deploy -g</span></span><br></pre></td></tr></table></figure></p><p><img src="/2018/06/22/build-blog/img/git_error.png" alt="git_error"></p><figure class="highlight jboss-cli"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">使用 hexo <span class="keyword">deploy</span> 命名部署到github失败，报上面的错误时，安装下面的插件即可解决:</span><br><span class="line">$ npm install hexo-deployer-git <span class="params">--save</span></span><br></pre></td></tr></table></figure><p>至此，Hexo+Github Pages构建个人博客网站已经基本完成了。可以通过网页访问自己的博客地址如：<a href="https://builder34.github.io" target="_blank" rel="noopener">https://builder34.github.io</a></p><h2 id="2-设置博客自定义域名"><a href="#2-设置博客自定义域名" class="headerlink" title="2.设置博客自定义域名"></a>2.设置博客自定义域名</h2><p>进入自己博客的repository仓库，通过类似如下的页面进行设置：<br><img src="/2018/06/22/build-blog/img/github_pages_settings_btn.png" alt="pages_settings"><br>进入了settings页面后，往下拉直到看到Github Pages模块：<br><img src="/2018/06/22/build-blog/img/github_pages_settings.png" alt="pages_settings"></p><p>所填的自定义域名是需要自己已经在万网上注册的了，并且如果勾选了 Enforce HTTPS 的话，你的域名是需要ssl证书的哟。</p><h2 id="3-注意事项"><a href="#3-注意事项" class="headerlink" title="3.注意事项"></a>3.注意事项</h2><h3 id="3-1-上传README-md并防止被渲染成文章"><a href="#3-1-上传README-md并防止被渲染成文章" class="headerlink" title="3.1 上传README.md并防止被渲染成文章"></a>3.1 上传README.md并防止被渲染成文章</h3><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">在博客根目录下，新建或编辑你的README.md文件</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> vim README.md</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> mv README.md ./sources</span></span><br><span class="line"><span class="meta">#</span><span class="bash">修改_config.yml文件,设置不渲染的文件</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> vim _config.yml</span></span><br><span class="line">skip_render: README.md</span><br></pre></td></tr></table></figure><h3 id="3-2-每次hexo-deploy后Github-Pages自定义域名会被重置的问题"><a href="#3-2-每次hexo-deploy后Github-Pages自定义域名会被重置的问题" class="headerlink" title="3.2 每次hexo deploy后Github Pages自定义域名会被重置的问题"></a>3.2 每次<code>hexo deploy</code>后Github Pages自定义域名会被重置的问题</h3><p>需要在sources目录下新建CNAME文件(注意为全大写无后缀的文件哦),文件内容为你需要映射到的自定义域名：<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> vim CNAME</span></span><br><span class="line">blog.monbuilder.top</span><br><span class="line"></span><br><span class="line"><span class="meta">$</span><span class="bash"> mv CNAME ./sources</span></span><br></pre></td></tr></table></figure></p><h2 id="4-写文章"><a href="#4-写文章" class="headerlink" title="4.写文章"></a>4.写文章</h2><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ hexo <span class="keyword">new</span> interview-java2018</span><br><span class="line">INFO Created: ~<span class="regexp">/Documents/</span>workspace<span class="regexp">/builder/</span>blog<span class="regexp">/builder34.github.io/</span><span class="keyword">source</span><span class="regexp">/_posts/i</span>nterview-java2018.md</span><br></pre></td></tr></table></figure><p>使用上面的命令创建文章，找到对应的md文件，使用你喜欢的markdown编辑器，尽情输出吧…<br>下面两个命令是，生成静态页面，再部署，然后就可以在你的博客网站上看到你新发布的文章了，哈哈！<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">$</span><span class="bash"> hexo g</span></span><br><span class="line"><span class="meta">$</span><span class="bash"> hexo d</span></span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<p>什么是Hexo？Hexo 是一个快速、简洁且高效的博客框架。Hexo 使用 Markdown（或其他渲染引擎）解析文章，在几秒内，即可利用靓丽的主题生成静态网页。堪称在座各位喜欢Markdown的优雅人士博客建站神器哟！</p>
<h2 id="1-Quick-Start"
</summary>
<category term="hexo博客" scheme="https://blog.monbuilder.top/categories/hexo%E5%8D%9A%E5%AE%A2/"/>
<category term="hexo" scheme="https://blog.monbuilder.top/tags/hexo/"/>
</entry>
</feed>