DDD CQRS架构和传统架构的优缺点比较   
最近几年,在DDD的领域,我们经常会看到CQRS架构的概念。我个人也写了一个ENode框架,专门用来实现这个架构。CQRS架构本身的思想其实非常简单,就是读写分离。是一个很好理解的思想。就像我们用MySQL数据库的主备,数据写到主,然后查询从备来查,主备数据的同步由MySQL数据库自己负责,这是一种数据库层面的读写分离。关于CQRS架构的介绍其实已经非常多了,大家可以自行百度或google。我今天主要想总结一下这个架构相对于传统架构(三层架构、DDD经典四层架构)在数据一致性、扩展性、可用性、伸缩性、性能这几个方面的异同,希望可以总结出一些优点和缺点,为大家在做架构选型时提供参考。

前言

CQRS架构由于本身只是一个读写分离的思想,实现方式多种多样。比如数据存储不分离,仅仅只是代码层面读写分离,也是CQRS的体现;然后数据存储的读写分离,C端负责数据存储,Q端负责数据查询,Q端的数据通过C端产生的Event来同步,这种也是CQRS架构的一种实现。今天我讨论的CQRS架构就是指这种实现。另外很重要的一点,C端我们还会引入Event Sourcing+In Memory这两种架构思想,我认为这两种思想和CQRS架构可以完美的结合,发挥CQRS这个架构的最大价值。

数据一致性

传统架构,数据一般是强一致性的,我们通常会使用数据库事务保证一次操作的所有数据修改都在一个数据库事务里,从而保证了数据的强一致性。在分布式的场景,我们也同样希望数据的强一致性,就是使用分布式事务。但是众所周知,分布式事务的难度、成本是非常高的,而且采用分布式事务的系统的吞吐量都会比较低,系统的可用性也会比较低。所以,很多时候,我们也会放弃数据的强一致性,而采用最终一致性;从CAP定理的角度来说,就是放弃一致性,选择可用性。

CQRS架构,则完全秉持最终一致性的理念。这种架构基于一个很重要的假设,就是用户看到的数据总是旧的。对于一个多用户操作的系统,这种现象很普遍。比如秒杀的场景,当你下单前,也许界面上你看到的商品数量是有的,但是当你下单的时候,系统提示商品卖完了。其实我们只要仔细想想,也确实如此。因为我们在界面上看到的数据是从数据库取出来的,一旦显示到界面上,就不会变了。但是很可能其他人已经修改了数据库中的数据。这种现象在大部分系统中,尤其是高并发的WEB系统,尤其常见。

所以,基于这样的假设,我们知道,即便我们的系统做到了数据的强一致性,用户还是很可能会看到旧的数据。所以,这就给我们设计架构提供了一个新的思路。我们能否这样做:我们只需要确保系统的一切添加、删除、修改操作所基于的数据是最新的,而查询的数据不必是最新的。这样就很自然的引出了CQRS架构了。C端数据保持最新、做到数据强一致;Q端数据不必最新,通过C端的事件异步更新即可。所以,基于这个思路,我们开始思考,如何具体的去实现CQ两端。看到这里,也许你还有一个疑问,就是为何C端的数据是必须要最新的?这个其实很容易理解,因为你要修改数据,那你可能会有一些修改的业务规则判断,如果你基于的数据不是最新的,那意味着判断就失去意义或者说不准确,所以基于老的数据所做的修改是没有意义的。

扩展性

传统架构,各个组件之间是强依赖,都是对象之间直接方法调用;而CQRS架构,则是事件驱动的思想;从微观的聚合根层面,传统架构是应用层通过过程式的代码协调多个聚合根一次性以事务的方式完成整个业务操作。而CQRS架构,则是以Saga的思想,通过事件驱动的方式,最终实现多个聚合根的交互。另外,CQRS架构的CQ两端也是通过事件的方式异步进行数据同步,也是事件驱动的一种体现。上升到架构层面,那前者就是SOA的思想,后者是EDA的思想。SOA是一个服务调用另一个服务完成服务之间的交互,服务之间紧耦合;EDA是一个组件订阅另一个组件的事件消息,根据事件信息更新组件自己的状态,所以EDA架构,每个组件都不会依赖其他的组件;组件之间仅仅通过topic产生关联,耦合性非常低。

上面说了两种架构的耦合性,显而易见,耦合性低的架构,扩展性必然好。因为SOA的思路,当我要加一个新功能时,需要修改原来的代码;比如原来A服务调用了B,C两个服务,后来我们想多调用一个服务D,则需要改A服务的逻辑;而EDA架构,我们不需要动现有的代码,原来有B,C两订阅者订阅A产生的消息,现在只需要增加一个新的消息订阅者D即可。

从CQRS的角度来说,也有一个非常明显的例子,就是Q端的扩展性。假设我们原来Q端只是使用数据库实现的,但是后来系统的访问量增大,数据库的更新太慢或者满足不了高并发的查询了,所以我们希望增加缓存来应对高并发的查询。那对CQRS架构来说很容易,我们只需要增加一个新的事件订阅者,用来更新缓存即可。应该说,我们可以随时方便的增加Q端的数据存储类型。数据库、缓存、搜索引擎、NoSQL、日志,等等。我们可以根据自己的业务场景,选择合适的Q端数据存储,实现快速查询的目的。这一切都归功于我们C端记录了所有模型变化的事件,当我们要增加一种新的View存储时,可以根据这些事件得到View存储的最新状态。这种扩展性在传统架构下是很难做到的。

可用性

可用性,无论是传统架构还是CQRS架构,都可以做到高可用,只要我们做到让我们的系统中每个节点都无单点即可。但是,相比之下,我觉得CQRS架构在可用性方面,我们可以有更多的回避余地和选择空间。

传统架构,因为读写没有分离,所以可用性要把读写合在一起综合考虑,难度会比较更大。因为传统架构,如果一个系统的高峰期的并发写入很大,比如为2W,并发读取也很大,比如为10W。那该系统必须优化到能同时支持这种高并发的写入和查询,否则系统就会在高峰时挂掉。这个就是基于同步调用思路的系统的缺点,没有一个东西去削峰填谷,保存瞬间多出来的请求,而必须让系统不管遇到多少请求,都必须能及时处理完,否则就会造成雪崩效应,造成系统瘫痪。但是一个系统,不会一直处在高峰,高峰可能只有半小时或1小时;但为了确保高峰时系统不挂掉,我们必须使用足够的硬件去支撑这个高峰。而大部分时候,都不需要这么高的硬件资源,所以会造成资源的浪费。所以,我们说基于同步调用、SOA思想的系统的实现成本是非常昂贵的。

而在CQRS架构下,因为CQRS架构把读和写分离了,所以可用性相当于被隔离在了两个部分去考虑。我们只需要考虑C端如何解决写的可用性,Q端如何解决读的可用性即可。C端解决可用性,我觉得是更加容易的,因为C端是消息驱动的。我们要做任何数据修改时,都会发送Command到分布式消息队列,然后后端消费者处理Command->产生领域事件->持久化事件->发布事件到分布式消息队列->最后事件被Q端消费。这个链路是消息驱动的。相比传统架构的直接服务方法调用,可用性要高很多。因为就算我们处理Command的后端消费者暂时挂了,也不会影响前端Controller发送Command,Controller依然可用。从这个角度来说,CQRS架构在数据修改上可用性要更高。不过你可能会说,要是分布式消息队列挂了呢?呵呵,对,这确实也是有可能的。但是一般分布式消息队列属于中间件,一般中间件都具有很高的可用性(支持集群和主备切换),所以相比我们的应用来说,可用性要高很多。另外,因为命令是先发送到分布式消息队列,这样就能充分利用分布式消息队列的优势:异步化、拉模式、削峰填谷、基于队列的水平扩展。这些特性可以保证即便前端Controller在高峰时瞬间发送大量的Command过来,也不会导致后端处理Command的应用挂掉,因为我们是根据自己的消费能力拉取Command。这点也是CQRS C端在可用性方面的优势,其实本质也是分布式消息队列带来的优势。所以,从这里我们可以体会到EDA架构(事件驱动架构)是非常有价值的,这个架构也体现了我们目前比较流行的Reactive Programming(响应式编程)的思想。

然后,对于Q端,应该说和传统架构没什么区别,因为都是要处理高并发的查询。这点以前怎么优化的,现在还是怎么优化。但是就像我上面可扩展性里强调的,CQRS架构可以更方便的提供更多的View存储,数据库、缓存、搜索引擎、NoSQL,而且这些存储的更新完全可以并行进行,互相不会拖累。理想的场景,我觉得应该是,如果你的应用要实现全文索引这种复杂查询,那可以在Q端使用搜索引擎,比如ElasticSearch;如果你的查询场景可以通过keyvalue这种数据结构满足,那我们可以在Q端使用Redis这种NoSql分布式缓存。总之,我认为CQRS架构,我们解决查询问题会比传统架构更加容易,因为我们选择更多了。但是你可能会说,我的场景只能用关系型数据库解决,且查询的并发也是非常高。那没办法了,唯一的办法就是分散查询IO,我们对数据库做分库分表,以及对数据库做一主多备,查询走备机。这点上,解决思路就是和传统架构一样了。

性能、伸缩性

本来想把性能和伸缩性分开写的,但是想想这两个其实有一定的关联,所以决定放在一起写。

伸缩性的意思是,当一个系统,在100人访问时,性能(吞吐量、响应时间)很不错,在100W人访问时性能也同样不错,这就是伸缩性。100人访问和100W人访问,对系统的压力显然是不同的。如果我们的系统,在架构上,能够做到通过简单的增加机器,就能提高系统的服务能力,那我们就可以说这种架构的伸缩性很强。那我们来想想传统架构和CQRS架构在性能和伸缩性上面的表现。

说到性能,大家一般会先思考一个系统的性能瓶颈在哪里。只要我们解决了性能瓶颈,那系统就意味着具有通过水平扩展来达到可伸缩的目的了(当然这里没有考虑数据存储的水平扩展)。所以,我们只要分析一下传统架构和CQRS架构的瓶颈点在哪里即可。

传统架构,瓶颈通常在底层数据库。然后我们一般的做法是,对于读:通常使用缓存就可以解决大部分查询问题;对于写:办法也有很多,比如分库分表,或者使用NoSQL,等等。比如阿里大量采用分库分表的方案,而且未来应该会全部使用高大上的OceanBase来替代分库分表的方案。通过分库分表,本来一台数据库服务器高峰时可能要承受10W的高并发写,如果我们把数据放到十台数据库服务器上,那每台机器只需要承担1W的写,相对于要承受10W的写,现在写1W就显得轻松很多了。所以,应该说数据存储对传统架构来说,也早已不再是瓶颈了。

传统架构一次数据修改的步骤是:1)从DB取出数据到内存;2)内存修改数据;3)更新数据回DB。总共涉及到2次数据库IO。

然后CQRS架构,CQ两端加起来所用的时间肯定比传统架构要多,因为CQRS架构最多有3次数据库IO,1)持久化命令;2)持久化事件;3)根据事件更新读库。为什么说最多?因为持久化命令这一步不是必须的,有一种场景是不需要持久化命令的。CQRS架构中持久化命令的目的是为了做幂等处理,即我们要防止同一个命令被处理两次。那哪一种场景下可以不需要持久化命令呢?就是当命令时在创建聚合根时,可以不需要持久化命令,因为创建聚合根所产生的事件的版本号总是为1,所以我们在持久化事件时根据事件版本号就能检测到这种重复。

所以,我们说,你要用CQRS架构,就必须要接受CQ数据的最终一致性,因为如果你以读库的更新完成为操作处理完成的话,那一次业务场景所用的时间很可能比传统架构要多。但是,如果我们以C端的处理为结束的话,则CQRS架构可能要快,因为C端可能只需要一次数据库IO。我觉得这里有一点很重要,对于CQRS架构,我们更加关注C端处理完成所用的时间;而Q端的处理稍微慢一点没关系,因为Q端只是供我们查看数据用的(最终一致性)。我们选择CQRS架构,就必须要接受Q端数据更新有一点点延迟的缺点,否则就不应该使用这种架构。所以,希望大家在根据你的业务场景做架构选型时一定要充分认识到这一点。

另外,上面再谈到数据一致性时提到,传统架构会使用事务来保证数据的强一致性;如果事务越复杂,那一次事务锁的表就越多,锁是系统伸缩性的大敌;而CQRS架构,一个命令只会修改一个聚合根,如果要修改多个聚合根,则通过Saga来实现。从而绕过了复杂事务的问题,通过最终一致性的思路做到了最大的并行和最少的并发,从而整体上提高系统的吞吐能力。

所以,总体来说,性能瓶颈方面,两种架构都能克服。而只要克服了性能瓶颈,那伸缩性就不是问题了(当然,这里我没有考虑数据丢失而带来的系统不可用的问题。这个问题是所有架构都无法回避的问题,唯一的解决办法就是数据冗余,这里不做展开了)。两者的瓶颈都在数据的持久化上,但是传统的架构因为大部分系统都是要存储数据到关系型数据库,所以只能自己采用分库分表的方案。而CQRS架构,如果我们只关注C端的瓶颈,由于C端要保存的东西很简单,就是命令和事件;如果你信的过一些成熟的NoSQL(我觉得使用文档性数据库如MongoDB这种比较适合存储命令和事件),且你也有足够的能力和经验去运维它们,那可以考虑使用NoSQL来持久化。如果你觉得NoSQL靠不住或者没办法完全掌控,那可以使用关系型数据库。但这样你也要付出努力,比如需要自己负责分库分表来保存命令和事件,因为命令和事件的数据量都是很大的。不过目前一些云服务如阿里云,已经提供了DRDS这种直接支持分库分表的数据库存储方案,极大的简化了我们存储命令和事件的成本。就我个人而言,我觉得我还是会采用分库分表的方案,原因很简单:确保数据可靠落地、成熟、可控,而且支持这种只读数据的落地,框架内置要支持分库分表也不是什么难事。所以,通过这个对比我们知道传统架构,我们必须使用分库分表(除非阿里这种高大上可以使用OceanBase);而CQRS架构,可以带给我们更多选择空间。因为持久化命令和事件是很简单的,它们都是不可修改的只读数据,且对kv存储友好,也可以选择文档型NoSQL,C端永远是新增数据,而没有修改或删除数据。最后,就是关于Q端的瓶颈,如果你Q端也是使用关系型数据库,那和传统架构一样,该怎么优化就怎么优化。而CQRS架构允许你使用其他的架构来实现Q,所以优化手段相对更多。

结束语

我觉得不论是传统架构还是CQRS架构,都是不错的架构。传统架构门槛低,懂的人也多,且因为大部分项目都没有什么大的并发写入量和数据量。所以应该说大部分项目,采用传统架构就OK了。但是通过本文的分析,大家也知道了,传统架构确实也有一些缺点,比如在扩展性、可用性、性能瓶颈的解决方案上,都比CQRS架构要弱一点。大家有其他意见,欢迎拍砖,交流才能进步,呵呵。所以,如果你的应用场景是高并发写、高并发读、大数据,且希望在扩展性、可用性、性能、可伸缩性上表现更优秀,我觉得可以尝试CQRS架构。但是还有一个问题,CQRS架构的门槛很高,我认为如果没有成熟的框架支持,很难使用。而目前据我了解,业界还没有很多成熟的CQRS框架,java平台有axon framework, jdon framework;.NET平台,ENode框架正在朝这个方向努力。所以,我想这也是为什么目前几乎没有使用CQRS架构的成熟案例的原因之一。另一个原因是使用CQRS架构,需要开发者对DDD有一定的了解,否则也很难实践,而DDD本身要理解没个几年也很难运用到实际。还有一个原因,CQRS架构的核心是非常依赖于高性能的分布式消息中间件,所以要选型一个高性能的分布式消息中间件也是一个门槛(java平台有RocketMQ),.NET平台我个人专门开发了一个分布式消息队列EQueue,呵呵。另外,如果没有成熟的CQRS框架的支持,那编码复杂度也会很复杂,比如Event Sourcing,消息重试,消息幂等处理,事件的顺序处理,并发控制,这些问题都不是那么容易搞定的。而如果有框架支持,由框架来帮我们搞定这些纯技术问题,开发人员只需要关注如何建模,实现领域模型,如何更新读库,如何实现查询,那使用CQRS架构才有可能,因为这样才可能比传统的架构开发更简单,且能获得很多CQRS架构所带来的好处。

paulwong 2017-02-19 08:39 发表评论

          Site Reliability Engineer (Systems) - Indeed - Andhra Pradesh   
We are experts in core infrastructure technologies like load balancers, HTTPd, Puppet, Tomcat, Memcached, RabbitMQ, Elasticsearch, MongoDB, and more....
From Indeed - Fri, 23 Jun 2017 07:16:27 GMT - View all Andhra Pradesh jobs
          This is what phylodiversity looks like   

Following on from earlier posts exploring how to map DNA barcodes and putting barcodes into GBIF it's time to think about taking advantage of what makes barcodes different from typical occurrence data. At present GBIF displays data as dots on a map (as do I in http://iphylo.org/~rpage/bold-map/). But barcodes come with a lot more information than that. I'm interested in exploring how we might measure and visualise biodiversity using just sequences.

Based on a talk by Zachary Tong (Going Organic - Genomic sequencing in Elasticsearch) I've started to play with n-gram searches on DNA barcodes using Elasticsearch, an open source search engine. The idea is that we break the DNA sequence into every possible "word" of length n (also called a k-mer or k-tuple, where k = n).

For example, for n = 5, the sequence GTATCGGTAACGAACTT would look like this:


GTATCGGTAACGAACTT

GTATC
TATCG
ATCGG
TCGGT
CGGTA
GGTAA
GTAAC
TAACG
AACGAA
ACGAAC
CGAACT
GAACTT

The sequence GTATCGGTAACGAACTT comes from Hajibabaei and Singer (2009) who discussed "Googling" DNA sequences using search engines (see also Kuksa and Pavlovic, 2009). If we index sequences in this way then we can do BLAST-like searches very quickly using Elasticsearch. This means it's feasible to take a DNA barcode and ask "what sequences look like this?" and return an answer qucikly enoigh for a user not to get bored waiting.

Another nice feature of Elasticsearch is that it supports geospatial queries, so we can ask for, say, all the barcodes in a particular region. Having got such a list, what we really want is not a list of sequences but a phylogenetic tree. Traditionally this can be a time consuming operation, we have to take the sequences, align them, then input that alignment into a tree building algorithm. Or do we?

There's growing interest in "alignment-free" phylogenetics, a phrase I'd heard but not really followed up. Yang and Zhang (2008) described an approach where every sequences is encoded as a vector of all possible k-tuples. For DNA sequences k = 5 there are 45 = 1024 possible combinations of the bases A, C, G, and T, so a sequence is represented as a vector with 1024 elements, each one is the frequency of the corresponding 5-tuple. The "distance" between two sequences is the mathematical distance between these vectors for the two sequences. Hence we no longer need to align the sequences being comapred, we simply chunk them into all "words" of 5 bases in length, and compare the frequencies of the 1024 different possible "words".

In their study Yang and Zhang (2008) found that:

We compared tuples of different sizes and found that tuple size 5 combines both performance speed and accuracy; tuples of shorter lengths contain less information and include more randomness; tuples of longer lengths contain more information and less random- ness, but the vector size expands exponentially and gets too large and computationally inefficient.

So we can use the same word size for both Elasticsearch indexing and for computing the distance matrix. We still need to create a tree, for which we could use something quick like neighbour-joining (NJ). This method is sufficiently quick to be available in Javascript and hence can be computed by a web browser (e.g., biosustain/neighbor-joining).

Putting this all together, I've built a rough-and-ready demo that takes some DNA barcodes, puts them on a map, then enables you to draw a box on a map and the demo will retrieve the DNA barcodes in that area, compute a distance matrix using 5-tuples, then build a NJ tree, all on the fly in your web browser.

Phylodiversity on the fly from Roderic Page on Vimeo.

This is all very crude, and I need to explore scalability (at the moment I limit the results to the first 200 DNA sequences found), but it's encouraging. I like the idea that, in principle, we could go to any part of the globe, ask "what's there?" and get back a phylogenetic tree for the DNA barcodes in that area.

This also means that we could start exploring phylogenetic diversity using DNA barcodes, as Faith & Baker (2006) wanted a decade ago:

...PD has been advocated as a way to make the best-possible use of the wealth of new data expected from large-scale DNA “barcoding” programs. This prospect raises interesting bio-informatics issues (discussed below), including how to link multiple sources of evidence for phylogenetic inference, and how to create a web-based linking of PD assessments to the barcode–of-life database (BoLD).

The phylogenetic diversity of an area is essentially the length of the tree of DNA barcodes, so if we build a tree we have a measure of diversity. Note that this contrasts with other approaches, such as Miraldo et al.'s "An Anthropocene map of genetic diversity" which measured genetic diversity within species but not between (!).

Practical issues

There are a bunch of practical issues to work through, such as how scalable it is to compute phylogenies using Javascript on the fly. For example, could we do something like generate a one degree by one degree grid of the Earth, take all the barcodes in each cell and compute a phylogeny for each cell? Could we do this in CouchDB? What about sampling, should we be taking a finite, random sample of sequences so that we try and avoid sampling bias?

There are also data management issues. I'm exploring downloading DNA barcodes, creating a Darwin Core Archive file using the Global Genome Biodiversity Network (GGBN) data standard, then converting the Darwin Core Archive into JSON and sending that to Elasticsearch. The reason for the intermediate step of creating the archive is so that we can edit the data, add missing geospatial informations, etc. I envisage having a set of archives, hosted say on GitHub. These archives could also be directly imported into GBIF, ready for the time that GBIF can handle genomic data.

References

  • Faith, D. P., & Baker, A. M. (2006). Phylogenetic diversity (PD) and biodiversity conservation: some bioinformatics challenges. Evol Bioinform Online. 2006; 2: 121–128. PMC2674678
  • Hajibabaei, M., & Singer, G. A. (2009). Googling DNA sequences on the World Wide Web. BMC Bioinformatics. Springer Nature. https://doi.org/10.1186/1471-2105-10-s14-s4
  • Kuksa, P., & Pavlovic, V. (2009). Efficient alignment-free DNA barcode analytics. BMC Bioinformatics. Springer Nature. https://doi.org/10.1186/1471-2105-10-s14-s9
  • Miraldo, A., Li, S., Borregaard, M. K., Florez-Rodriguez, A., Gopalakrishnan, S., Rizvanovic, M., … Nogues-Bravo, D. (2016, September 29). An Anthropocene map of genetic diversity. Science. American Association for the Advancement of Science (AAAS). https://doi.org/10.1126/science.aaf4381
  • Yang, K., & Zhang, L. (2008, January 10). Performance comparison between k-tuple distance and four model-based distances in phylogenetic tree reconstruction. Nucleic Acids Research. Oxford University Press (OUP). https://doi.org/10.1093/nar/gkn075

          Containers, microservices, and data   

DockerSome notes on containers, microservices, and data. The idea of packaging software into portable containers and running them either locally or in the cloud is very attractive (see Docker). Some use cases I'm interested in exploring.

Microservices

In Towards a biodiversity knowledge graph (doi:10.3897/rio.2.e8767) I listed a number of services that are essentially self contained, such as name parsers, reconciliation tools, resolvers, etc. Each of these could be packaged up and made into containers.

Databases

We can use containers to package database servers, such as CouchDB, ElasticSearch, and triple stores. Using containers means we don't need to go through the hassle of installing the software locally. Interested in RDF? Spin up a triple store, play with it, then switch it off if you decide it's not for you. If it proves useful, you can move it to the cloud and scale up (e.g., sloppy.io).

Data

A final use case is to put individual datasets in a container. For exmaple, imagine that we have a large Darwin Core Archive. We can distribute this as a simple zip file, but you can't do much with this unless you have code to parse Darwin Core. But imagine we combine that dataset with a simpel visualisation tool, such as VESpeR (see doi:10.1016/j.ecoinf.2014.08.004). Users interested in the data could then play with the data without the overhead of installing specialist software. In a sense, the data becomes an app.


          大数据搜索选开源还是商业软件?ElasticSearch 对比 Splunk   

本文就架构,功能,产品线,概念等方面就ElasticSearch和Splunk做了一下全方位的对比,希望能够大家在制定大数据搜索方案的时候有所帮助。

大数据搜索选开源还是商业软件?ElasticSearch 对比 Splunk,首发于文章 - 伯乐在线


          Using elasticsearch to Speed Up Filtering   
At work, we have a number of REST services that are responsible for most of the content you see on IGN. For example, all our reviews and features are stored in an articles service. Data about games (release date, publisher, … Continue reading
          Site Reliability Engineer (Systems) - Indeed - Andhra Pradesh   
We are experts in core infrastructure technologies like load balancers, HTTPd, Puppet, Tomcat, Memcached, RabbitMQ, Elasticsearch, MongoDB, and more....
From Indeed - Fri, 23 Jun 2017 07:16:27 GMT - View all Andhra Pradesh jobs
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Systems Engineer - Zipwhip, Inc. - Seattle, WA   
Riak, Cassandra, MongoDB, ElasticSearch, etc. We are looking for an Systems Engineer for our DevOps team that will help support and grow our systems and network...
From Indeed - Fri, 12 May 2017 22:07:41 GMT - View all Seattle, WA jobs
          僕のチームの開発ルール   

僕のチームの開発ルール、なんていうと大げさだし内容があってない感あるけど触発されて書いてみる。ただの思い出の殴り書き。

CVS

java servlet で IE をうにょうにょ動かす web? システムを作っていた。その頃はプロジェクトで CVS を使っていた。SVN 1.0 が出てはいたが 1.1 はまだだった。そもそも存在を知らなかったと思う。

wiki を使っているチームもあったが、基本的には excel に書いて共通フォルダに置いていたような気がする。ある程度の仕様書は web 上に置かれていた。最新化されることもなく情報共有の認識も薄かったのでそれぞれでメモを持っていた。客先(開発室)にいたこともあって充分なネット環境がなく、air edge で 256kbps はええ!トルネード速くなるけど画像が荒い!といった時期だった。

Eclipse の CVS 連携で表示される差分が見やすくて非常に良かった。差分表示機能だけ取り出せないものかと試行錯誤してた気がするけど、今思えば winmerge とか使えばよかったんじゃないだろうか。知識も検索力も乏しい時代だったので何かと時間がかかっていたように思う。

最近は Eclipse がバカにされている?風潮があるけど、上位バージョンで使える機能を除けば昨今の visual studio より扱いやすかった印象。たまにメモリ不足で落ちることを除けばバックグラウンドでビルドしてくれるしアウトラインがポップアップで出るし起動するまでに dll 読み込みすぎてアレとかなかったし(これは構成の問題だけど)。jsp を書き換えて IE を F5 でリロードして即反映されるのが楽だった。js も書き換えれば即反映だったし。生産性は悪くなかったと思うけど業務知識も開発スキルも低くて黒歴史が量産された。あの頃は若かった問題。

当時は IE6 なのもあってパフォーマンスに悩まされ続けたけど最近は PC の性能も上がりまくって爆速になってたりする。

Vimmer になったのもその頃で ViPlugin が便利だった。途中から vrapper に切り替えてさらに便利になった。

CVS と Redmine

相変わらず CVS で開発してユーザーごとにタグを打つとか移動するとかしていた。ファイル単位に。それはそれで小回りがきいて激しく困ったことはなかった。バージョンもそんなに別れてなかったし。

この頃はほぼ一人で開発とメンテをしていたのだけど、機能追加やバグ修正に伴って記録が全くないのはマズイだろうと一人 Redmine を始める。自分がいなくなっても経緯とかはここを見てもらえばなんとかなるだろうというのもあった。

数人が入っては抜けてを繰り返し自分も抜けて無事に引き渡しができた。見ることはないけど今も稼働しているので十分に役立った。

SVN

別プロジェクトが始まって SVN 運用がはじまった。開発中は何も考えずコミットしていけばよかったけど、リリースのためのブランチがきられると地獄がはじまった。コミット単位のマージがめんどくさいし分岐したブランチの統合は何回か試みが行われた後は戻そうと言い出す人はいなくなった。マージ担当が徹夜でコンフリクトを直した結果、マージ元と先が逆になってて早朝に破棄されてたのは胸が熱くなった。お疲れ。

元々資源管理に関する運用を考えるということに対して全体的に希薄だったのがよくなかった。マージすればいいんでしょ、ぐらいの軽い気持ちだったように思う。

SVN だけで複数の案件を並行して開発するのは無理に感じたので、僕は git-svn を使っていた。他の人は歯を食いしばっていた、ように思っていたけど git に切り替わるとSVN の方が良かった(分かり易かった)と言う声をかなり聞くのでこだわってたのはごく少数だったのかもしれない。

SVN + Redmine

課題管理を何でするか?となった時に excel 管理は嫌だ!と言い通して(駄々をこねて)導入。チームとして Redmine を使う事になる。最初はみんな慣れてなくて右往左往するか何も書き込まない。継続的な啓蒙活動と Redmine おじさん活動を行う。こまめにチケットに反応しているとようやく要領が分かったようで浸透していった。ついでにチケット上での喧嘩がちょいちょい勃発していたけど思春期みたいなもので一定期間過ぎると無くなっていった。

最初は windows server に SVN を立てていたのだけど、別の windows 機に立ててる Redmine がリポジトリの内容を引っ張れなくて残念だった。リビジョン番号はチケットに書いているけど、その修正内容は Redmine で見れないのでリリース系にマージされてるのかを調べるのが手間でメリットが半減していた。

チケット番号と SVN のログからコミットされているかを付き合わせて一覧化するスクリプトを書いて、出荷のたびにエクセルでリリースブランチにマージしてるのかしてないのかを書き込んでもらう運用。

その後、SVN のサーバが変わったタイミングで取れるようになったのでチケットとコミットが紐づけることができるようになった。チケットを開けばマージ状況はもちろん diff も見れるようになったのでいろんなことが捗るようになった。ようやく一人前になれた気がした。

必要に応じて Redmine 本体をいじったり plugin を書いたりして運用中。

最近は親子関係のチケットが量産されているのが気になっているが、自発的にチケットおこしやコメントをすることが当たり前になっている。「それチケットに登録されてる?」「チケットの番号は?」という会話が普通にされている。

SVN + Redmine + Jenkins

開発が進むにつれてビルドエラーが目立つようになる。うっかりコミット漏れもあるがビルドが通るかの確認もなくコミットされてるケースもあった。自分のところだけならまだいいけど、他チームにも迷惑をかけてしまうことも多々あった。ちょうど良いタイミングで Jenkins のセミナーがあったので受講。ググれと言われればそれまでだけれど、順序立てて教えてもらえるのは分かりやすくてありがたかった。すぐに自チーム用の Jenkins を立てて日に5回程度ビルドして注意を促した。同時に他チーム起因によるエラーを見つけては連絡していた。だいたいみんな「自分のところではビルド通ってますけどね」と言う。

Redmine と同じサーバに Jenkins も入れていたのでビルド中は Redmine が重いなぁと感じることがあったけど、最近はパフォーマンスアップされたのかビルド中でも特に感じなくなった。Jenkins でいい感じに並列ビルドできず時間がまぁまぁかかるのが難点。SSD にしたい。プロジェクト構成的に並列ビルドが厳しいのもなんとかしたいけど手遅れ感がある。

git

プロジェクト開始当初も git にするかという話はあったのだけど勉強コストがあるのと、今から変えると大混乱することが目に見えていたので先送りにされていた。

開発に 1 つの区切りができたのもあり、SVN だと辛いことが多いのもあって git への移行が決定される。言い出しっぺなところもあるけど、蓋を開けてみると git を知っている人が数人しかおらず、github を使っていた自分が一番詳しかったようで旗振り役になる。

詳しかったと言っても一人プロジェクトだったので、複数人でやるときの merge や rebase 運用についてはいろいろ調べながら試行錯誤で進めた。yokohama.vim でも相談させてもらった。運用はしているけど、それが一般的なのか間違ってるのかが分からないので、外の人に直接話を聞ける場があるっていうのは非常にありがたかった。

Github はお金的な問題でダメ。Gitbucket か GitLab かで迷ったけれど GitLab を採用。 Gitbucket ではフックがまだ使えなかったのが大きな理由だった。試用を兼ねて立てているけど、jar 一個で動くお手軽感はすげーなと思う。

GitLab は良いタイミングでパフォーマンスチューニングが活発になっていたのも大きかった。ネットでは重いと評判だったし自分で立てて検証してみても確かに厳しいなという感触だった。それがアップデートの度にいい感じに改善していた。現状でも重いなと思うことはしばしばあるけど “まだ” 許容範囲。

もう 1 つタイミングが良かったなと思うのが git for windows の改善。コマンド入力が苦手な人が多すぎるのは置いとくとして、問題なく使えている。packman を使ってパッケージを入れられるのも良かった。このタイミングで cygwin をようやく捨てることができた。長い間本当にお世話になった。ck も。

GitLab をセットアップし、定期的にアップデートして、Ruby でフックを作った。Redmine の plugin に次ぐ、業務で Ruby のお仕事。

ブランチ戦略を考えたり、wiki に手順や困った時の tips などなど沢山書き込んで情報発信しまくった。当初は使ってなかったけど merge request を紹介して運用に乗せた。今は当たり前に使っていてソースレビューも行われている。

現在の運用はほぼ手を離れていて担当の人に任せてある状態。何かあればフォローする程度になった。

次は

Elasticsearch などを使って Redmine やアプリログの可視化をしたいなと思っている。けどプロジェクトのピークがずっと続いててなかなか手をつけられてない。

まとめ

某 naoya さんがスクルトをかける方が好きだという表現をしていたけど、僕もそっちがあってるっぽい。レベルが違うので同じとか言うのも恥ずかしいけど。 自分でコードを書くし設計したりもするけど、開発スピードを上げるための活動やリリースの改善をした結果、助かる人がいるとか喜んでくれる人がいるとやる気がでる。 本来やるべきこととはズレたボランティアベースな所が多々ある。活動として給料が出るといいんだけど、本来やるべきところを後ろにずらしてやってたりするのでトントンか。他にも効率よく開発できるようにリポジトリの掃除や整備も地味にやってたりするんだけど、なかなか評価されない所ではある。自分からアピールしたりもしないけど。みんなで効率悪いことやってると気になってしまうので仕方がない。

今年度は (も) 色々あって激動の年になりそうだけど、適度に頑張る。体が元気ならなんとでもなる。


          PHP Developer / Proactive Appointments / Guildford, Surrey, United Kingdom   
Proactive Appointments/Guildford, Surrey, United Kingdom

Our client are looking for a talented PHP Developer to join their expanding Development Team(comprising of PHP and Front End developers).

As a key member of an agile team, you will be responsible for producing clean, scalable and

maintainable code in order to grow our bespoke eCommerce platform.

Responsibilities

Work as part of an agile development team

Create clean, maintainable, well thought out code

Design, deliver, test, and maintain new features

Build upon and improve the existing code base

Essential Skills and Experience:

OO PHP 5.6+

Linux Shell

MySQL

MVC Framework experience (eg Zend Framework 2, Symfony, Laravel etc.)

Version control (preferably Git)

Strong written and verbal communication skills

Comfortable operating in a loosely structured and agile organisation

Self-starter

Desirable Skills and Experience

Composer

HTML5

CSS

Javascript

Redis

Mongo

Elasticsearch

Node.js

Scrum

PHPUnit

Due to the volume of applications received for positions, it will not be possible to respond to all applications and only applicants who are considered suitable for interview will be contacted.

Proactive Appointments Limited operates as an employment agency and employment business and is an equal opportunities organisation.

Employment Type: Permanent

Apply To Job
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          你应该知道的18个大数据工具   

你应该知道的18个大数据工具

在当今的数字革命浪潮中,大数据成为公司企业分析客户行为和提供个性化定制服务的有力工具,大数据切切实实地帮助这些公司进行交叉销售,提高客户体验,并带来更多的利润。

随着大数据市场的稳步发展,越来越多的公司开始部署大数据驱动战略。

Apache Hadoop是目前最成熟的大数据分析工具,但是市场上也不乏其他优秀的大数据工具。目前市场上有数千种工具能够帮你节约时间和成本,带你从全新的角度洞察你所在的行业。

以下介绍18种功能实用的大数据工具:

Avro:由Doug Cutting公司研发,可用于编码Hadoop文件模式的数据序列化。

Cassandra:一种分布式的开源数据库。可用于处理商品服务器在提供高可用性服务时产生的大量分布式数据。这是一种非关系型数据库(NoSQL)解决方案,最初由Facebook主导研发。

目前很多公司组织都在使用这一数据库,如Netflix,Cisco,Twitter。

Drill:一种开源分布式系统,用于大规模数据集的交互分析。Drill与谷歌的Dremel系统类似,由Apache公司管理运行。

Elasticsearch:Apache Lucene开发的开源搜索引擎。Elasticsearch是基于Java的系统,可以实现高速搜索,支持你的数据搜索工作。

Flume:使用网络服务器、应用服务器和移动服务器的数据来填充Hadoop的大数据应用框架,是数据源和Hadoop之间的一种连接纽带。

HCatalog:是针对Apache Hadoop的集中元数据管理和分享服务。可以通过它集中查看Hadoop集群中的所有数据,并可以在不知道数据在集群中存储位置的情况下,通过Pig和 Hive等多种工具处理所有数据元素。

Impala: 使用与Apache Hive相同的元数据,SQL语法(Hive SQL),ODBC驱动程序和用户界面(HueBeeswax),直接帮助您对存储在HDFS或HBase中的Apache Hadoop数据进行快速的交互式SQL查询。

它为批量导向或实时查询提供了一个方便操作的统一平台。

JSON:今天的许多非关系型数据库(NoSQL)都以JSON(javascript对象符号)格式存储数据,这些格式在Web开发人员中很受欢迎。

Kafka:这是种分布式“发布——订阅”的消息传送系统,它能够提供一种解决方案,帮助处理所有数据流活动,并在消费者网站上处理这些数据。

这种类型的数据(包括页面查看数据,搜索数据和其他用户操作数据)是当前社交网络的关键组成部分。

MongoDB:是一个在开源概念指导下开发出来的面向文档的非关系型数据库(NoSQL)。它具有完整的索引支持,同时可以灵活地对任何属性进行索引,并在不影响功能的情况下进行横向扩容。

Neo4j:是一个图形数据库,与关系数据库相比,性能提升高达1000多倍或更高。

Oozie:一种工作流程处理系统,可以让用户自定义不同语言编写的一系列工作,如Map Reduce,Pig 和 Hive。它还可以实现不同工作项目之间的智能连接,Oozie还支持用户指定依赖关系。

Pig:是由雅虎开发的基于Hadoop的一种语言,对于用户来说,学习起来相对简单,且Pig擅长处理非常深入且非常长的数据管道(data pipeline)。

Storm:是一种免费的进行实时分布式计算的开源系统。通过Storm,用户可以非常轻松的在能够进行实时处理操作的范围内,对非结构化数据流进行可靠处理。

系统具有容错特性,支持几乎所有编程语言,当然最常用的语言还是Java。Storm最初是Apache家族的一个分支,现在已被Twitter收购。

Tableau:是一种主要关注商业智能的数据可视化工具。用户无需编程,就可以利用Tableau创建地图,条形图,散点图等可视化图像。

他们最近发布了一个Web连接器,允许用户直接连接数据库或应用程序界面(API),从而使用户能够在进行可视化项目时获取实时数据。

ZooKeeper:为大型分布式系统提供集中配置和开放代码名称注册的服务。

每天大数据技术领域都会涌现出大量新的大数据相关工具,要想学会使用每个工具是非常困难且没有意义的。挑选几个你能够熟练使用的工具,并不断学习技术知识,才是最好的方式。

作者:Deeplearner H 译者:灯塔大数据



你应该知道的18个大数据工具

          Which environments and versions does Dynatrace support?   
<div class="callout"> <p>In addition to software and hardware requirements, this page lists all environments, technologies, and versions that can be monitored with Dynatrace. This list is continuously growing, so please check back often.</p> <p>Use the <strong>On this page</strong> links on the right to jump to the information you&#x2019;re looking for.</p> </div> <h2 id="infrastructure-and-cloud">Infrastructure and cloud</h2><h3 id="operating-systems">Operating systems</h3><h4 id="dynatrace-oneagent">Dynatrace OneAgent</h4><table> <thead> <tr> <th>Operating system</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Windows Server</td> <td>2008, 2008 R2, 2012, 2012 R2, 2016</td> </tr> <tr> <td>Windows</td> <td>7, 8, 8.1, 10 (64 bit)</td> </tr> <tr> <td>Red Hat Enterprise Linux</td> <td>5 (without SELinux), 6.1+</td> </tr> <tr> <td>CentOS</td> <td>6+</td> </tr> <tr> <td>Ubuntu</td> <td>12.04+</td> </tr> <tr> <td>openSUSE</td> <td>12+</td> </tr> <tr> <td>Debian</td> <td>6+</td> </tr> <tr> <td>Fedora</td> <td>20+</td> </tr> <tr> <td>IBM AIX for Java and IBM HTTP Server (currently in Beta)</td> <td>7.1, 7.2</td> </tr> <tr> <td>Amazon Linux</td> </tr> <tr> <td>CoreOS</td> <td></td> </tr> <tr> <td>SUSE Enterprise Linux</td> <td>12+</td> </tr> <tr> <td>Compatible with any standards-compliant x86_64 distribution based on glibc 2.5+</td> </tr> </tbody> </table> <h4 id="security-gateway">Security Gateway</h4><p>64 bit physical or virtual host (no container) required</p> <table> <thead> <tr> <th>Operating system</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Windows Server</td> <td>2008, 2008 R2, 2012, 2012 R2</td> </tr> <tr> <td>Windows</td> <td>7, 8, 8.1,10</td> </tr> <tr> <td>Red Hat Enterprise Linux</td> <td>6+</td> </tr> <tr> <td>Ubuntu</td> <td>12.04+</td> </tr> <tr> <td>openSUSE</td> <td>13</td> </tr> </tbody> </table> <h4 id="dynatrace-managed-server1">Dynatrace Managed Server<sup>1</sup></h4><p>64 bit physical or virtual host (no container) required</p> <table> <thead> <tr> <th>Operating system</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Red Hat Enterprise Linux</td> <td>6+</td> </tr> <tr> <td>CentOS</td> <td>6+</td> </tr> <tr> <td>Ubuntu</td> <td>12.04+</td> </tr> <tr> <td>openSUSE</td> <td>12.3+</td> </tr> <tr> <td>SUSE Enterprise Linux</td> </tr> </tbody> </table> <p><sup>1</sup>Dynatrace Managed installations only</p> <h3 id="networks">Networks</h3><h4 id="network-interface-requirements">Network interface requirements</h4><ul> <li>IEEE 802.3 Ethernet, IEEE 802.11 Wireless LAN, OpenVZ virtual network device (venet)</li> <li>Physical or virtual NIC with assigned (no link-local) IP address</li> <li>Virtual Ethernet bridge interface is not supported</li> <li>Network interface bonding is supported</li> </ul> <h4 id="protocols">Protocols</h4><ul> <li>TCP/IPv4</li> <li>TCP/IPv6 without extension headers</li> </ul> <h3 id="virtualization">Virtualization</h3><h4 id="monitoring">Monitoring</h4><table> <thead> <tr> <th>VMware</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>vCenter server</td> <td>5.x, 6.0, 6.5</td> </tr> <tr> <td>ESXi host</td> <td>5.x, 6.0, 6.5</td> </tr> </tbody> </table> <h4 id="detection">Detection</h4><ul> <li>Xen</li> <li>KVM</li> <li>Hyper-V</li> </ul> <h3 id="cloud">Cloud</h3><h4 id="monitoring-1">Monitoring</h4><table> <thead> <tr> <th>Amazon Web Services (AWS)</th> </tr> </thead> <tbody> <tr> <td>Elastic Compute Cloud (EC2)</td> </tr> <tr> <td>Elastic Block Store (EBS)</td> </tr> <tr> <td>Elastic Load Balancing (ELB)</td> </tr> <tr> <td>Relational Database Service (RDS)</td> </tr> <tr> <td>Simple Storage Service (S3)</td> </tr> <tr> <td>DynamoDB</td> </tr> <tr> <td>Lambda</td> </tr> </tbody> </table> <h4 id="detection-1">Detection</h4><ul> <li>OpenStack</li> <li>Oracle Cloud</li> <li>Microsoft Azure</li> </ul> <h3 id="paas">PaaS</h3><h4 id="azure">Azure</h4><ul> <li><a href="/infrastructure-monitoring/paas/how-do-i-monitor-microsoft-azure-web-apps/">Azure Web Sites extension</a></li> <li><a href="/infrastructure-monitoring/paas/how-do-i-deploy-oneagent-to-azure-compute-vms/">Azure VM extension</a></li> <li><a href="/infrastructure-monitoring/paas/how-to-monitor-azure-service-fabric-applications/">Azure Service Fabric</a></li> </ul> <h4 id="cloud-foundry">Cloud Foundry</h4><table> <thead> <tr> <th>Feature</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Java buildpack</td> <td>3.11+</td> </tr> <tr> <td>IBM WebSphere Liberty buildpack</td> <td>v3.9-20170419-1403+</td> </tr> <tr> <td>PHP buildpack</td> <td>v4.3.34+</td> </tr> <tr> <td>Staticfile buildpack</td> <td>v1.4.6+</td> </tr> </tbody> </table> <h3 id="docker">Docker</h3><table> <thead> <tr> <th>Feature</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Deep monitoring</td> <td>1.6+ (32 and 64 bit) glibc or musl-libc required</td> </tr> <tr> <td>Docker containers plugin</td> <td>1.8, 1.9, 1.10, 1.11, 1.12, 1.13 RC2, 1.13.1</td> </tr> <tr> <td>Kubernetes</td> <td>1.3.1 - 1.5.3 supported by all OneAgent versions, 1.5.4 - 1.6.2 supported by OneAgent versions 121+</td> </tr> </tbody> </table> <p><strong>Note</strong>: Dynatrace doesn&#x2019;t support Docker monitoring for Windows when OneAgent is installed on a Windows host.</p> <p><a name="applications-services--databases"></a></p> <h2 id="applications-services--databases">Applications, services, &amp; databases</h2><p>Looking for <a href="#net">.NET</a>, <a href="#php">PHP</a>, or <a href="#nodejs">Node.js</a>?</p> <h3 id="java-and-scala">Java and Scala</h3><table> <thead> <tr> <th>Virtual machines</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Oracle Hotspot</td> <td>5, 6, 7, 8</td> </tr> <tr> <td>Oracle JRockit</td> <td>6</td> </tr> <tr> <td>OpenJDK</td> <td>6, 7, 8</td> </tr> <tr> <td>IBM J9</td> <td>6, 7, 8</td> </tr> <tr> <td>SAP JVM</td> <td>6, 7, 8</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Application servers</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Apache Tomcat</td> <td>6, 7, 8</td> </tr> <tr> <td>Apache TomEE</td> <td>1.6</td> </tr> <tr> <td>Jetty</td> <td>7, 8, 9</td> </tr> <tr> <td>IBM WebSphere Application Server</td> <td>8, 8.5, 8.5.5</td> </tr> <tr> <td>Oracle WebLogic</td> <td>11g, 12c</td> </tr> <tr> <td>JBoss Application Server</td> <td>6, 7</td> </tr> <tr> <td>JBoss Enterprise Application Platform</td> <td>6</td> </tr> <tr> <td>WildFly</td> <td>8, 9, 10</td> </tr> <tr> <td>Glassfish (Linux)</td> <td>3.1.x, 4.x</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Web frameworks</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Servlet Engine</td> <td>2.3, 2.4, 2.5, 3.0, 3.1</td> </tr> <tr> <td>Grails (in servlet container)</td> <td>3.x</td> </tr> <tr> <td>Play Framework</td> <td>2.2, 2.3, 2.4</td> </tr> <tr> <td>Ning Asynchronous HTTP Client</td> <td>1.8, 1.9</td> </tr> <tr> <td>Netty</td> <td>3.10, 4.x</td> </tr> <tr> <td>ElasticSearch</td> <td>1.7, 2.0, 2.1, 2.2</td> </tr> <tr> <td>Vert.x</td> <td>2, 3</td> </tr> <tr> <td>Apache HttpClient</td> <td>3.1, 4.x</td> </tr> <tr> <td>Apache HttpAsyncClient</td> <td>4.0.x, 4.1.x</td> </tr> <tr> <td>Java HttpUrlConnection</td> </tr> <tr> <td>webMethods</td> <td>8.2, 9.0, 9.5</td> </tr> <tr> <td>Mule HTTP Listener</td> <td>3.5, 3.6, 3.7, 3.8</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Web services</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Play WS API</td> <td>2.2, 2.3, 2.4</td> </tr> <tr> <td>JAX-WS</td> <td>2.x</td> </tr> <tr> <td>Apache Axis2</td> <td>1.6</td> </tr> <tr> <td>Apache CXF</td> <td>2.x, 3.x</td> </tr> <tr> <td>Jersey</td> <td>1.x, 2.x</td> </tr> <tr> <td>Spring Web Services</td> <td>2.x</td> </tr> <tr> <td>JBossWS (JBoss AS)</td> <td>4.x (Jboss 7.1)</td> </tr> <tr> <td>JBossWS (Wildfly)</td> <td>4.x (Wildfly 8), 5.x (Wildfly 8, 9, 10)</td> </tr> <tr> <td>REST web services via WINK framework</td> <td>1.2, 1.4</td> </tr> <tr> <td>Restlet WS</td> <td>2.1 - 2.3</td> </tr> <tr> <td>Hessian web services</td> <td>2.1, 3.1, 4.0</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Database frameworks</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td><a href="https://docs.mongodb.org/ecosystem/drivers/java/">MongoDB driver</a></td> <td>2.x, 3.x</td> </tr> <tr> <td><a href="http://mongodb.github.io/mongo-java-driver/3.0/driver-async/">MongoDB Async driver</a></td> <td>3.x</td> </tr> <tr> <td>JDBC</td> <td>all</td> </tr> <tr> <td>Cassandra (Thrift)</td> <td>2.x</td> </tr> <tr> <td>Jedis Redis</td> <td>2.x</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Messaging libraries</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>JMS</td> <td>1.1, 2.0</td> </tr> <tr> <td>Apache ActiveMQ</td> <td>4, 5</td> </tr> <tr> <td>HornetQ</td> <td>2.2, 2.3, 2.4</td> </tr> <tr> <td>RabbitMQ</td> <td>3.x</td> </tr> <tr> <td>IBM WebSphere MQ client</td> <td>7.x, 8.x</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Remoting services</th> </tr> </thead> <tbody> <tr> <td>RMI-IIOP</td> </tr> <tr> <td>RMI-JRMP</td> </tr> <tr> <td>JBoss Remoting</td> </tr> </tbody> </table> <h3 id="net">.NET</h3><p>Dynatrace fully supports .NET applications written in C#. Limited support for .NET applications written in other languages is available, though not explicitly tested.</p> <p>.NET and CLR version combinations:</p> <table> <thead> <tr> <th>.NET Framework</th> <th>CLR</th> </tr> </thead> <tbody> <tr> <td>3.5 SP1</td> <td>Full CLR 2.0</td> </tr> <tr> <td>4.0</td> <td>Full CLR 4.0</td> </tr> <tr> <td>4.5</td> <td>Full CLR 4.0</td> </tr> <tr> <td>4.5.1</td> <td>Full CLR 4.0</td> </tr> <tr> <td>4.5.2</td> <td>Full CLR 4.0</td> </tr> <tr> <td>4.6</td> <td>Full CLR 4.0</td> </tr> <tr> <td>4.6.1</td> <td>Full CLR 4.0</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Web and web service frameworks</th> </tr> </thead> <tbody> <tr> <td>ASP.NET</td> </tr> <tr> <td>ASP.NET Core</td> </tr> <tr> <td>ASP.NET Owin/Katana</td> </tr> <tr> <td>WCF</td> </tr> <tr> <td>Service Fabric Service remoting</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Database frameworks</th> </tr> </thead> <tbody> <tr> <td>ADO.NET</td> </tr> </tbody> </table> <h3 id="nodejs">Node.js</h3><ul> <li>Versions: 0.10, 0.12, 4, 5, 6, 7</li> </ul> <table> <thead> <tr> <th>Web frameworks</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>HTTP, HTTPS</td> <td>built-in</td> </tr> <tr> <td>Express</td> <td>3, 4</td> </tr> <tr> <td><a href="https://www.npmjs.com/package/restify">restify</a></td> <td>2.x</td> </tr> </tbody> </table> <table> <thead> <tr> <th>Database frameworks</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td><a href="https://www.npmjs.com/package/mongodb">MongoDB driver</a></td> <td>1.4.x and 2.x</td> </tr> <tr> <td>Redis</td> <td></td> </tr> <tr> <td>Memcached</td> <td></td> </tr> <tr> <td>SQLite3 (context passing only)</td> </tr> </tbody> </table> <h3 id="php">PHP</h3><ul> <li>Linux (mod_php, FastCGI or PHP-FPM) required</li> <li>Versions: 5.3, 5.4, 5.5, 5.6, 7</li> </ul> <table> <thead> <tr> <th>Database frameworks</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td><a href="http://php.net/manual/en/book.pdo.php">PDO</a></td> <td>built-in</td> </tr> <tr> <td><a href="http://php.net/manual/en/set.mysqlinfo.php">mysql, mysqli</a></td> <td>built-in</td> </tr> </tbody> </table> <ul> <li>All PHP frameworks and PHP-based application platforms, including:<ul> <li>Magento</li> <li>Drupal</li> <li>Joomla</li> <li>WordPress</li> </ul> </li> </ul> <h2 id="web-servers">Web servers</h2><table> <thead> <tr> <th>Server</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Apache HTTP Server</td> <td>2.0 (deprecated), 2.2, 2.4</td> </tr> <tr> <td>Microsoft IIS</td> <td>7.0, 7.5, 8.0, 8.5, 10</td> </tr> <tr> <td>IBM HTTP Server</td> <td>6.0, 7.0, 8.0, 8.5, 9.0</td> </tr> <tr> <td>Oracle HTTP Server</td> <td>11g, 12c</td> </tr> <tr> <td>Nginx</td> <td>Any Linux build that contains debug symbols of the following versions 1.4, 1.6, 1.7, 1.8, 1.9.0-1.9.15, 1.10.0-1.10.3, 1.11.0-1.11.13, 1.12.0, 1.13.0, plus R1-R12. Alternatively, you can use a dedicated <a href="/get-started/reference/supported-nginx-binaries/">supported binary</a> listed below. Note that these binaries don&#x2019;t require debug symbols.</td> </tr> </tbody> </table> <h4 id="list-of-linux-distributions-for-nginx-binaries">List of Linux distributions for Nginx binaries</h4><ul> <li><a href="http://archive.ubuntu.com/">http://archive.ubuntu.com/</a></li> <li><a href="http://archive.webtatic.com/">http://archive.webtatic.com/</a></li> <li><a href="http://dl.fedoraproject.org/">http://dl.fedoraproject.org/</a></li> <li><a href="http://download.opensuse.org/">http://download.opensuse.org/</a></li> <li><a href="http://ftp.debian.org/">http://ftp.debian.org/</a></li> <li><a href="http://nginx.org/">http://nginx.org/</a></li> <li><a href="http://packages.eu-west-1.amazonaws.com/">http://packages.eu-west-1.amazonaws.com/</a></li> <li><a href="http://ppa.launchpad.net/">http://ppa.launchpad.net/</a></li> <li><a href="http://security.debian.org/">http://security.debian.org/</a></li> <li><a href="http://www2.atomicorp.com/">http://www2.atomicorp.com/</a></li> <li><a href="https://archives.dotdeb.org/">https://archives.dotdeb.org/</a></li> <li><a href="https://copr-be.cloud.fedoraproject.org/">https://copr-be.cloud.fedoraproject.org/</a> </li> <li><a href="https://oss-binaries.phusionpassenger.com/">https://oss-binaries.phusionpassenger.com/</a></li> <li><a href="https://packages.dotdeb.org/">https://packages.dotdeb.org/</a></li> <li><a href="https://packages.gitlab.com/">https://packages.gitlab.com/</a></li> <li><a href="https://packages.graylog2.org/">https://packages.graylog2.org/</a></li> <li><a href="https://plus-pkgs.nginx.com/">https://plus-pkgs.nginx.com/</a></li> <li><a href="https://pulp.inuits.eu/">https://pulp.inuits.eu/</a></li> <li><a href="https://repo.webtatic.com/">https://repo.webtatic.com/</a></li> </ul> <h2 id="plugins">Plugins</h2><p><strong>Dynatrace plugins only run on physical or virtual hosts</strong>, not within containers.</p> <table> <thead> <tr> <th>Technology</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>MySQL</td> <td>5.7</td> </tr> <tr> <td>Microsoft SQL Server</td> <td>2008, 2012, 2014</td> </tr> <tr> <td>PostgreSQL</td> <td>9.1+</td> </tr> <tr> <td>MongoDB</td> <td>2.6+</td> </tr> <tr> <td>HAProxy</td> <td>Daemon-mode only (-D option) 1.4, 1.5, 1.6</td> </tr> <tr> <td>Redis</td> <td>2.8+</td> </tr> <tr> <td>Memcached</td> <td>1.4.24+</td> </tr> <tr> <td>RabbitMQ</td> <td>3.4.0+</td> </tr> <tr> <td>Elasticsearch</td> <td>2.3+</td> </tr> <tr> <td>.NET CLR</td> <td></td> </tr> <tr> <td>CouchDB</td> <td>1.5.0+</td> </tr> <tr> <td>Couchbase</td> <td>4.3+</td> </tr> <tr> <td>Hadoop</td> <td>2.4.1+</td> </tr> <tr> <td>Spark</td> <td>1.6</td> </tr> <tr> <td>Cassandra</td> <td>2.0+</td> </tr> <tr> <td>ActiveMQ</td> <td>5.8.0+</td> </tr> </tbody> </table> <h2 id="real-user-monitoring-and-web-checks">Real user monitoring and web checks</h2><h3 id="real-user-monitoring-auto-injection">Real user monitoring auto-injection</h3><ul> <li>Java servlet-based web applications</li> <li>Apache HTTP Server</li> <li>IBM HTTP Server</li> <li>Oracle HTTP Server</li> <li>Microsoft IIS</li> <li>Nginx</li> <li>Node.js</li> </ul> <h3 id="real-user-monitoring-javascript-frameworks">Real user monitoring JavaScript frameworks</h3><table> <thead> <tr> <th>Framework</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>AngularJS</td> <td>1.0 - 1.5</td> </tr> <tr> <td>AngularJS 2</td> <td>2</td> </tr> <tr> <td>Dojo</td> <td>1.6.1 - 1.10.4</td> </tr> <tr> <td>Ext JS</td> <td>3.4, 4, 5, 6.0</td> </tr> <tr> <td>Sencha Touch</td> <td>2.0 - 2.4</td> </tr> <tr> <td>ICEfaces</td> <td>1.8, 2, 3</td> </tr> <tr> <td>jQuery ( Backbone.js )</td> <td>1.3 - 1.11, 2.0 - 2.2, 3.0</td> </tr> <tr> <td>MooTools</td> <td>1.4.6 - 1.5</td> </tr> <tr> <td>Prototype</td> <td>1.6, 1.7</td> </tr> <tr> <td>XMLHttpRequest (XHR)</td> </tr> </tbody> </table> <h3 id="mobile-real-user-monitoring">Mobile real user monitoring</h3><table> <thead> <tr> <th>Operating System</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Android</td> <td>2.2+ (API 8+)</td> </tr> <tr> <td>iOS</td> <td>6+</td> </tr> </tbody> </table> <h2 id="browsers">Browsers</h2><h3 id="real-users">Real users</h3><table> <thead> <tr> <th>Browser</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Microsoft Internet Explorer</td> <td>8+</td> </tr> <tr> <td>Microsoft Edge</td> <td>Latest version</td> </tr> <tr> <td>Mozilla Firefox</td> <td>3 latest versions</td> </tr> <tr> <td>Google Chrome</td> <td>3 latest versions (desktop and mobile)</td> </tr> <tr> <td>Safari</td> <td>3 latest versions (OS X and iOS)</td> </tr> <tr> <td>Android Browser</td> <td>3 latest versions</td> </tr> <tr> <td>Opera</td> <td>2 latest versions</td> </tr> <tr> <td>All modern browsers with JavaScript and cookies enabled</td> </tr> </tbody> </table> <h3 id="synthetic-users">Synthetic users</h3><table> <thead> <tr> <th>Feature</th> <th>Support</th> </tr> </thead> <tbody> <tr> <td>Recorder</td> <td>Google Chrome (latest version, backwards compatible)</td> </tr> <tr> <td>Web check</td> <td>Google Chrome 53</td> </tr> </tbody> </table> <h3 id="dynatrace-web-ui">Dynatrace web UI</h3><table> <thead> <tr> <th>Browser</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Microsoft Edge</td> <td>Latest version (desktop and mobile)</td> </tr> <tr> <td>Mozilla Firefox</td> <td>Latest version (desktop)</td> </tr> <tr> <td>Google Chrome</td> <td>Latest version (desktop and mobile)</td> </tr> <tr> <td>Safari</td> <td>10+ (OS X and iOS)</td> </tr> <tr> <td>Microsoft Internet Explorer</td> <td>11 (desktop)</td> </tr> </tbody> </table> <ul> <li>Please note that Microsoft Internet Explorer provides significantly lower performance than other listed browsers. It&#x2019;s recommended that you use an alternative browser.</li> </ul> <h2 id="dynatrace-messaging--alerting">Dynatrace messaging &amp; alerting</h2><h3 id="integrations">Integrations</h3><ul> <li>PagerDuty</li> <li>HipChat API version 2</li> <li>Slack</li> <li>VictorOps</li> </ul> <h3 id="dynatrace-mobile-app">Dynatrace Mobile app</h3><table> <thead> <tr> <th>Operating System</th> <th>Versions</th> </tr> </thead> <tbody> <tr> <td>Android</td> <td>4.1+</td> </tr> <tr> <td>iOS</td> <td>8+</td> </tr> <tr> <td>Windows Phone</td> <td>8.1+</td> </tr> <tr> <td>Windows</td> <td>10</td> </tr> </tbody> </table>
          Alexander Naumov: openSUSE Conference 2017   

В конце прошлого месяца разработчики openSUSE снова собрались в Нюрнберге, чтобы обсудить дальнейшее развитие дистрибутива и просто пообщаться и весело провести время вместе. В последние пару лет проект стал очень быстро меняться. К проекту присоединилось очень много новых людей. Новые идеи, которые они приносят, находят свое место в новых проектах, разрабатываемых для openSUSE. Даже используя все информационные каналы проекта openSUSE, тяжело уследить за всеми новшествами, включаемыми в проект, и просто меняющейся стратегией совета. Поэтому вопрос о посещении конференции, тем более, если вы мейнтейнер, долгого размышления не потребует 🙂


Конференция была открыта докладом Матиаса о LiMux. Краткая история проекта, его взлет, надежды и его падение… Явный пример того, на сколько опасны могут быть политики, пользующиеся властью для удовлетворения собственных предпочтений, и закрывающие глаза на интересы и желания населения. В настоящий момент я как раз работаю над этим проектом в администриции Мюнхена.

На конференции я снова встретился с Дмитрием, занимающимся проектом invis и продвижением Free Software решений на базе 1C в Германии. Беседовали о русскоязычном сообществе openSUSE, и о причинах его столь плачевного состояния. Я очень надеюсь, что сообществу получится преодалеть существующие проблемы и оно снова возьмет курс на развитие, как было в 2008-2010 годах.

Познакомился с Денисом Кондратенко из Киева и его женой. Очень приятная пара. Денис рассказывал о Ceph и EKG в openSUSE, а также о методах обработки метаданных в Elasticsearch. Мой личный опыт использования Elasticsearch ограничивается только BigData/Hadoop, поэтому и тут удалось узнать что-то новое.

Много новых иновационных идей было услышано от Ричарда. Он уже давно стал openSUSE evangelist’ом; его доклады об OBS или openQA можно услышать практически на каждой европейской Free Software конференции, посвященной GNU/Linux. Я всем советую посмотреть его доклад о Containerised Application.


Одним из спонсоров конференции в этом году стал fedora project. Fedora уже не первый год использует нашу систему openQA для автоматического тестирования linux-систем, а сотрудники RedHat уже во второй раз читают свои доклады на openSUSE Conference. В этом году это был доклад How semantic analysis of C and C++ ELF binaries can be used to analyze ABI changes, в прошлом году это были Enforcement of a system-wide crypto policy и Testing complex software in CI.

В общем, как и обычно, конферениция оставила приятное впечатление (пару фоток можно найти тут). И хотя в этом году она длилась всего 3 дня вместо 5, как в прошлом, информации для размышления я пролучил придостаточно. Её посещение в этом году не стоило мне ничего. Спасибо за это TSP. В следующем году она пройдет в Праге, куда я планирую поехать с семьей. Возможно там я встречусь и с теми, кто читает сейчас эти стоки 😉



          文章: eBay里Elasticsearch集群的生命周期   

eBay的“Elasticsearch即服务”平台名叫Pronto,它可以为各种不同的搜索用例提供全面管理下的Elasticsearch集群。平台现在管理着至少35个集群,支持多个数据中心部署。为了实现对Elasticsearch集群的最优化管理,这篇文章针对定义一个集群生命周期的方方面面都提供了指导。

By Sudeep Kumar
          (USA-CO-Boulder) Sr. C++/C# Engineer   
**About IHS Markit** IHS Markit harnesses deep sources of information, analytics and expertise to forge solutions for the industries and markets that drive global economies. Our company partners with clients in business, finance and government to provide the unrivaled insights and perspectives that lead to well-informed, confident decisions. We call this The New Intelligence. IHS Markit serves more than 50,000 key customers in more than 140 countries, including 80 percent of the Fortune Global 500. We help decision makers apply higher-level thinking to daily tasks and strategic issues across a host of industries and disciplines including energy, finance, automotive, engineering, technology, maritime and trade, aerospace and defense, chemical, and economics and country risk. Headquartered in London, IHS Markit (Nasdaq: INFO) is committed to sustainable, profitable growth. _______________________________________________________________ **Markit Digital is looking for talented, intellectual developers who have a passion for solving difficult problems. We need software engineers who can help us develop and maintain systems that are always online and can handle hundreds of thousands of transactions per second.** **We write code primarily in 1) C++ using features from C++11 and C++14, and 2) C#. We are looking for engineers who stay up-to-date with the latest advances in technology, whether that means C++, C#, Java, compilers, hardware, or operating systems. You are passionate about technology and can act as an ambassador to other teams within Markit to keep our systems moving forward.** A good candidate will have a solid understanding of software design patterns and architectural best practices. Much of our time is spent focusing on developing new capabilities or reworking and enhancing legacy capabilities, necessitating strong design skills which will lead to robust solutions. We spend a lot of time interfacing with various system components which require understanding of networking, application configuration, and server configuration. While you may not be directly responsible for operations activities, you will need to understand systems in order to interact with DevOps and infrastructure teams . We make use of various SQL and no-SQL solutions, so familiarity with MS SQL Server, Cassandra, Hadoop, ElasticSearch, etc. are a plus. We hold regular code reviews to ensure our software is at the highest quality and so that we can all learn from each other. We regularly collaborate via whiteboard design sessions and virtual sessions. We constantly challenge our own assumptions and never accept “because that is the way we’ve always done it” as a justification. As a developer, your work will used by millions of real people internationally and by some of the largest financial institutions in the world. **_Here are some examples of the volumes of data we regularly handle to whet your appetite:_** • 130 million document search queries per week across an indexed universe of 98 million documents • 3.2 million market data quotes processed per second • 400 million identification symbols contained in our cross referencing system. **Some other things we look for:** • A good sense of humor, eagerness to learn, and a drive to be successful. • Real-time event processing background and high-throughput optimization skills. • Experience with concurrent / parallel execution. • Experience working with the big data in a high-speed, high-availability environment. • CMake and TeamCity knowledge helpful. • ElasticSearch experience is a bonus. • Background in the financial markets, economic theory, or statistics is a plus. We are proud to be an EEO/AA employer M/F/Disability/Veterans. Please refer to the **_EEO is the Law Poster & Supplement, Right to Work and Pay Transparency Policy._** **_IHS is a participant in E-Verify (see links below)_** . We maintain a drug-free workplace and perform pre-employment drug testing. EEO is the Law at http://www1.eeoc.gov/employers/upload/eeoc_self_print_poster.pdf EEO is the Law Supplement at http://www.dol.gov/ofccp/regs/compliance/posters/pdf/OFCCP_EEO_Supplement_Final_JRF_QA_508c.pdf Right to Work at http://www.uscis.gov/sites/default/files/USCIS/Verification/E-Verify/E-Verify_Native_Documents/OSC_Right_to_Work_Poster.pdf Pay Transparency Policy at https://www.dol.gov/ofccp/pdf/pay-transp_formattedESQA508c.pdf E-Verify at https://e-verify.uscis.gov/emp/media/resourcesContents/EverifyPosterEnglish.pdf IHS Markit endeavors to make our career sites accessible to any and all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please contact Accommodation@ihs.com or call 303-858-6891. **_This contact information (email and phone) is intended for accommodation requests only._** Unfortunately we are unable to accept resumes or provide information about application status through the email address above. **_Resumes are only accepted through the online application process, and only qualified candidates will receive consideration and follow-up._**
          Scaling WP Queries with Elasticsearch   
WordPress.com is big. Currently, there are: 18.6 billion pages views/month 409 million unique visitors/month 56 million new posts/month 68 million new comments/month All of this […]
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Installation Walkthrough: OCI 7.3 and the Data Warehouse - Part 2 of 3   
Part 2: Installing OnCommand Insight 7.3 Server

Installation Pre-requisites

- The pagefile should be > 8GB.

Before the Setup Wizard

1) Open an Administrative Command Prompt

Image: Getting an Administrative Command Prompt

2) Navigate to where the MSI is stored, and from the prompt run> SANscreenServer-x64-7.3.0-1032.msi

Setup Wizard

1) Welcome to the OnCommand Insight 7.3.0 Setup Wizard

Click Next

2) End-User License Agreement

Tick ‘I accept the terms in the License Agreement’
Click Next

3) Customer Information

Input: Customer Name
Input: Site Name
Browse to desired installation location.
Click Next

Image: OCI 7.3.0 Setup - Customer Information

4) Customer Information - Configure NetApp ASUP

Configure NetApp ASUP as required:
Database: Backup without Performance Data / No Database Backup / Backup with Performance Data
Send Method: HTTP / HTTPS / FTP / Email
Logs: Extended Logs / Base Logs / No Logs
Click Next

Image: OCI 7.3.0 Setup - Configure NetApp ASUP (defaults)

5) Configure Server

OnCommand Insight Portal Port (HTTP): 80
OnCommand Insight Portal Port (HTTPS): 443
Internal Database Port (SQL): 3306
Click Next

Image: OCI 7.3.0 Setup - Configure Server (defaults)

6) Configure Elasticsearch

Elasticsearch Transport Port: 9310
Elasticsearch REST Port: 9200
Click Next

Image: OCI 7.3.0 Setup - Configure Elasticsearch (defaults)

7) Ready to install OnCommand Insight 7.3.0

Click Install

8) Completed the OnCommand Insight 7.3.0 Setup Wizard

Click Finish

Image: OCI 7.3.0 Setup - Completed

Result

All being well, you should have a desktop shortcut for ‘OnCommand Insight Client’ that points to something like the below:
O:\OCISVR\SANscreen\java\bin\javaws.exe -J https://OCISVR.lab.priv:443/client/app/http-client.jnlp

The following services should be started (check in services.msc):
- Elasticsearch
- MySQL
- SANscreen Acq
- SANscreen Server

Logging in

Java UI: Double-click the link on the desktop, accept any Java prompts, and default credentials are:

User name: admin
Password: admin123

Web UI: Connect to https://FQDN:443

*Can also login as oadmin / ******

Licensing

From the Web UI at https://FQDN/uiserver/#admin/settings/license
Click on the Update License button
Copy and paste in your license
Click Save

Next Steps to Configure OCI

From the ‘OnCommand Insight 7.3 Configuration and Administration Guide’ and page 11 ‘Setting up Insight’:

- Install your Insight licenses.
- Set up your data sources in Insight.
- Set up user accounts.
- Configure your email.
- Define your SNMP, email, or syslog notifications if needed.
- Enable automatic weekly backups of your Insight database.
- Perform any advanced configuration steps required, including defining annotations and thresholds.


          Tech Round Up 22nd June 2017   
It’s been a long time since I did a Tech Round Up/Research Links & Info post (16th April 2015). There’s been an accumulation of tech stuff in my personal inbox recently, time to clear them down.

3 sections below:
- DevOps Stuff
- NetApp HCI
- NetApp Random Stuff


DevOps Stuff

1) Play with docker classroom

One of my colleagues sent me this awesome link:

2) DevOps Tools

Image: Periodic Table of DevOps Tools

The same colleague sent me this link to Periodic Table of DevOps Tools. Whilst it’s very cool and all clickable with excellent links, I’m not quite sure why it’s presented as a Periodic Table. If you’re boring like me and would like to see simple lists of the DevOps Tools, here they are:

SCM (Source Code Management):
Open Source: Git, Gitlab, Subversion, Mercurial
Freemium: Github, Bitbucket
Enterprise: ISPW

Database Mgmt:
Open Source: Liquibase
Enterprise: DBmaestro, Redgate, Datical, Delphix, Idera

Build:
Open Source: Maven, Grunt, MSBuild, Gradle, Gulp, Make, Rake, ANT, Broccoli, CMake
Free: SBT, Packer

Testing:
Open Source: FitNesse, Mocha, Cucumber.js, Gatling, Jasmine
Free: Cucumber, Junit, Selenium, JMeter, Karma, Qunit, TestNG

Repo Mgmt:
Open Source: NPM, Artifactory, Nexus
Free: DockerHub

CI (Continuous Integration):
Open Source: Jenkins, Continuum, Travis CI, CruiseControl
Freemium: Codeship, TeamCity, Shippable, Continua CI, CircleCI
Paid: Bamboo, Visual Studio, Solano CI

Config / Provisioning:
Open Source: Ansible, Vagrant, Consul, Salt, Bcfg2, CFEngine
Free: Terraform
Enterprise: Chef, Puppet, BladeLogic

Containerization:
Open Source: Docker, RKT, Mesos, Kubernetes, Nomad
Free: Swarm

Cloud / IaaS / Paas:
Open Source: OpenStack
Freemium: Amazon Web Services, Heroku
Paid: Azure
Enterprise: Google Cloud Platform, Rackspace, OpenShift

Release Mgmt:
Enterprise: XL Release, UrbanCode Release, BMC Release Process, CA Release Automation, Automic, Plutora Release, Serena Release

Collaboration:
Freemium: Trello, HipChat, Slack, Flowdock
Paid: Team Foundation, Jira, Pivotal Tracker
Enterprise: ServiceNow

BI / Monitoring:
Open Source: Kibana, Nagios, Zabbix, Elasticsearch
Freemium: New Relic, AppDynamics
Enterprise: Dynatrace, Datadog

Logging:
Open Source: Logstash
Freemium: Logentries, Sumo Logic
Enterprise: Splunk

Security:
Open Source: Snort, Tripwire
Enterprise: Fortify


NetApp HCI

NetApp released their NetApp HCI (Hyper Converged Infrastructure) product on 5thJune 2017. This looks like a great product and I hope to get my hands on some in the future. It is based on their SolidFire platform. A few links:



And especially check out the following videos:
NetApp HCI Demo (You can be up and running in about 30 minutes!)


NetApp Random Stuff

1) NetAppDocs-Lite for Customer Collection

It’s no secret that NetApp have an awesome tool for creating 7-Mode and ONTAP as-built documentation called NetAppDocs. The full tool is only available to NetApp partners and internal staff. Customers can download the NetAppDocs-Lite version to collect the XML for NetApp partners/internal staff to process into as built documentation. Check out this link:


2) One specific UNIX account is failing to access its NAS data on NetApp Clustered ONTAP 8.3.2 post 7 to C transition

We had a UNIX account that’s failing to access its data post 7 to C transition. Just one UNIX account was affected. Running::>*

diag secd authentication translate -node NODE -vserver VSERVER -uid UID

Results in “Unable to retrieve UNIX username for UID”. It turned out that the following 7-Mode option was disabled (it’s hidden by default):

options nfs.authsys.extended_groups_ns.enable

But the SVM had it configured. Disabling it fixed the problem::>

vserver nfs modify -vserver VSERVER -auth-sys-extended-groups disabled

3) CN1610 Cluster Switches Hung to Management (Telnet, SSH and Serial)

I’ve known the CN1610’s become unmanageable a few times; telnet, SSH, serial - none of those work. There is a way to break-in to the Linux shell and collect logs for troubleshooting (so don’t just reboot the switch and accept it as a quirk). Understandably this information is only available to NetApp partners and internal staff. This is the link:


4) End of Engineering Support for Clustered Data ONTAP 8.3.x on 31st October 2017

A date to make a note of if your ONTAP systems are still on 8.3.x. Engineering Support ends on the 31st of October 2017 (I understand that the end of limited support is 30th April 2019.) This is a great excuse to get your systems upgraded. Check out this link:


What does end of ‘Engineering Support’ mean?

- Requests for bug fixes on the Data ONTAP 8.3.x release family will no longer be accepted. Fixes for issues that are found on AFF, FAS, and V-Series systems that run Data ONTAP 8.3.x will be delivered only in patch derivatives of ONTAP 9.x.
- Interoperability testing with Data ONTAP 8.3.x and updates to the NetApp Interoperability Matrix Tool will stop.
- Manufacturing and SAN qualification PVRs that are specific to the Data ONTAP 8.3.x release family will not be accepted.
- For systems that cannot be upgraded to ONTAP 9.x, specifically FAS3240 and V3240, for the remainder of the customer’s current support entitlement, NetApp will provide critical updates if NetApp determines that they are necessary and commercially reasonable.

5) NetApp is No.1 in current quarter with 32% quarterly revenue share in EMEA AFA (All Flash Array) market


Image: EMEA - Top 5 Vendors AFA Dashboard

6) NetApp Data Fabric Solution for Cloud Backup Resources

Link to help plan, prepare and execute an ONTAP to AltaVault Snapshot strategy, managed and catalogued using SnapCenter software:

          es v5.4.3   

相変わらず量が多すぎる

Release v5.4.3 · elastic/elasticsearch · GitHub


          有起有伏,使用 Go 一年的体验分享   

我们公司 Mobile Jazz 从一个内部试验性项目开始使用 Go。如公司名暗示的那样,我们是开发移动应用的。

在发布一个应用给公众后,我们很快意识到我们缺失一个工具来检查用户实际发生的情况以及他们是如何与应用交互的 - 如果有任何问题或者 bug 的报告,这将会相当方便。

现在有几款工具声称能在这个方面帮助开发者,但是没有一个能完全满足要求,因此我们决定自己构建一个。我们开始创建一组基础的脚本,如今它很快进化成了完整的工具,称为 Bugfender

由于这最初是一个实验,我们决定使用一种新的趋势技术。对学习以及持续教育的热爱是 Mobile Jazz 的核心价值的之一,因此我们决定使用 Go 构建。这是一个由 Google 开发的相对较新的编程语言。它是编程世界的的新玩家,已经有许多受尊敬的开发者对它赞不绝口。

一年后,这个实验变成了一个初创项目,我们拥有了一个已经帮助了来自全世界的数以千计的开发者的令人难以置信的工具。我们的服务器每天处理来自 700 万台设备的超过 200GB 的数据。

在使用 Go 一年之后,我们想要分享我们将一个小小的实验变成处理百万日志的生产服务器的一些想法和经验。

Go 生态系统

公司中没有人有使用 Go 的经验。Bugfender 是我们第一次深入这个语言。

学习基本上相当直接的。我们之前在 C/C++/Java/Objective-C/PHP 的经验让我们学习 Go 相当快,并且在几天内就开始开发了。当然会有一些新的和不常见的东西需要学习,包括 GOPATH 还有如何处理包,但这在我们的预期之内。

几天之内,我们意识到即使是一个以简化为设计目的的语言,Go 也是非常强大的。它能够做任何现代编程语言应该能做的事:能够处理 JSON、服务器之间通讯甚至访问数据库也没问题(并且只需要几行代码)。

在构建一个服务器时,你应该首先决定是否使用任何第三方库或者框架。对于 Bugfender,我们决定使用:

Martini

Martini 是一个强大的 Go 的 web 框架。我们开始这个实验时,它是一个很棒的解决方案,至今也是,我们还没遇到任何问题。然而如果我们今天再次开始这个实验的话,我们会选择一个不同的框架,因为 Martini 不在维护了。

我们还试验了 Iris(我们目前的最爱)还有 Gin。Gin 是 Martini 继任者,并且迁移到这上能让我们重用已有的代码。

在过去的一年中,我们意识到 Go 的标准库是非常强大的,你不必依靠一个臃肿的 web 框架来构建一个服务器。最好在特定任务上使用专门的高性能库。

Iris 是我们目前最喜欢的,并且将来我们将使用它重写服务来替代 Martini/Gin,但这目前并不是优先的。

修改: 在讨论了 Iris 的各个方面之后,我们意识到 Iris 或许不是最好的选择。如果我们决定重写我们的 web 组件,我们或许会研究其他的选择,我们欢迎你的建议。

Gorm

有些人喜欢 ORM,而有些人则不喜欢。我们决定使用 ORM,更确切地说是 GORM。我们的实现只针对 web 前端,对于日志提取 API 仍然继续使用手工优化的 SQL。在一开始,我们确实很喜欢它,但是随着时间的推移,我们开始发现问题,并且我们很快将它从代码中完全移除,并且使用 sqlx 这个标准 SQL 库。

GORM 的一个主要问题是 Go 的生态系统。作为一个新语言,自我们开始开发产品以来 Go 已经有很多新版本。在这些新版本中的一些改变并不向后兼容,因此要使用最新的库版本,我们要经常重写已有代码并检查我们为解决版本问题所做的 hack。

上述这两个库是大多数 web 服务的主要组件,因此做一个好的选择很重要,因为将来更改会很困难,并且会影响你服务器的性能。

第三方服务

在创建一个实际使用的产品的另外一个重要方面是考虑库、第三方服务和工具的可用性。在这方面,Go 还缺乏成熟度,大多数公司还没有提供 Go 库,因此你或许需要依赖其他人写的不能一直保证质量的库。

比如,对于使用 RedisElasticSearch 有很好的库,但是对于其他服务比如 Mixpanel 或者 Stripe 还没有好的。

我们建议在使用 Go 之前事先检查对于你需要的产品是否有好的库可用。

我们在 Go 的包管理系统上也遇到了很多问题。它处理版本的方式远没有达到最好,并且在过去的一年中,我们在不同的团队成员之间使用同一个库的不同版本上遇到了很多问题。然而,最近要归功于 Go 新支持的 vendor 包特性,除了 gopkg.in 服务外,这个问题基本被解决了。

开发者工具

由于 Go 是一门相对新的语言,你或许发现相比其他成熟的语言像 Java,它可用的开发工具并不很好。当我们开始 Bugfender 时,使用任何 IDE 都很困难,似乎没有 IDE 支持 Go。但是在过去的一年中,随着 IntelliJVisual Studio Code Go 插件的引入,这一切改善了很多。

最后看下其他的 Go 工具,调试器并不很好,而分析器甚至更糟,因此有时调试你的代码或者尝试优化它会很困难。

前往生产

这确实是 Go 最好的东西之一,如果你想要部署一些东西到生产环境中,你只需要构建你的二进制并发送到服务器中,没有依赖,不需要安装额外的软件,你只需要能在服务器中运行二进制文件就行。

如果你习惯于处理那些需要包管理器或者需要小心你使用的语言解释器的语言,用 Go 工作会感到很高兴。

我们对 Go 的稳定性也很满意,因为服务器似乎从没有崩溃过。我们在发送大量数据给 Go Routines 时遇到过一个问题,但是我们几乎没见到任何崩溃。注意:如果你需要发送大量数据给 Go Routine,你需要小心堆溢出。

如果你对性能感兴趣,我们没法与其他语言相比较,因为我们从零开始使用 Go,但是对于我们处理的数据量,我们感觉性能是非常好的,我们绝对不能如此轻松地使用 PHP 处理同等数量的请求。

总结

在过去的一年中,我们对 Go 的感觉起起伏伏。最初我们是兴奋的,但是在实验变成真实的产品后我们开始发现问题。我们几次考虑过用 Java 完全重写,但是目前为止,仍旧使用的是 Go,并且过去的一年中, Go 生态已经有了很大的提升,这简化了我们的工作。

如果你想要使用 Go 构建你的产品,你可以保证它可以工作,但是你确实需要小心一件事:可以雇佣的开发者。硅谷中只有很少的高级 Go 开发者,并且在其他地方寻找也是一件非常困难的任务。

via: https://bugfender.com/one-year-using-go

作者:ALEIX VENTAYOL 译者:geekpi 校对:wxy

来源:LCTT 

>>>【评论有礼】6月6日-30日评论每日更新的“新闻资讯和软件更新资讯”,评论点赞数超过 20 的可登上每周更新的“源资讯”和“软件周刊”两大栏目,点赞数超过 50 的还将获得 5 活跃积分奖励和开源中国定制好礼。详情


          GitLab 9.3.2 发布,代码托管平台   

GitLab 9.3.2 已发布,该版本修复了 9.3.1 版本中的一些回归和错误。详细如下:

  • CE/EE: Resolve "Submitting reply to existing diff discussion using Cmd/Ctrl+Enter submits twice and refreshes page" (!12352)

  • CE/EE: Resolve "Unable to access edit comment from dropdown menu in certain screen sizes" (!12421)

  • CE/EE: Revert "Annotate" to "Blame" (!12401)

  • CE/EE: Fix optional arguments for POST :id/variables (!12474)

  • CE/EE: Fixes problem with the Action Buttons on a Label Item Line (!12473)

  • CE/EE: Fix application error when Project#last_activity_at is nil (!12443)

  • CE/EE: Truncate long job names in environment view; wrap author to next line (!12455)

  • CE/EE: Bump premailer-rails gem to 1.9.7 and its dependencies to prevent network retrieval of assets (!12456)

  • CE/EE: Fix bug where Service created_at time was used instead of deployment time. (!12395)

  • CE/EE: Update mmap2 gem to fix missing symbol error that happens on some installations from source (!12492)

  • EE: Fix gitlab:check Rake task when Elasticsearch used (!2278)

  • EE: Resolve "LDAP group link select dropdown error" (!2277)

  • Omnibus: Update gitlab-monitor to 1.8.0 (!1685)

发布主页

下载地址


          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Systems Engineer - Zipwhip, Inc. - Seattle, WA   
Riak, Cassandra, MongoDB, ElasticSearch, etc. We are looking for an Systems Engineer for our DevOps team that will help support and grow our systems and network...
From Indeed - Fri, 12 May 2017 22:07:41 GMT - View all Seattle, WA jobs
          Comentario en Elasticsearch and Kibana on EMR Hadoop cluster por IT Tech   
Perfect, thanks so much for this Hernan... exactly what I needed!
          Comentario en Elasticsearch and Kibana on EMR Hadoop cluster por hvivani   
Hi, Yes, I have bootstrap actions for EMR 4.X and 5.x Please use these: s3://awssupportdatasvcs.com/bootstrap-actions/elasticsearch/elasticsearch_install.4.0.0.rb s3://awssupportdatasvcs.com/bootstrap-actions/elasticsearch/kibananginx_install.4.0.0.rb Both should work on 4.x and 5.x Let me know if you experience any issues. Regards, Hernan
          Comentario en Elasticsearch and Kibana on EMR Hadoop cluster por IT Tech   
Hi there, I'm interested in using these bootstrap scripts for AWS EMR 4.x and 5.0, but it appears they only support AMI version 3.x. Do you know if it's possible to run these scripts on later versions of EMR, or how to get Elasticsearch and Kibana installed manually? Thanks!
          Site Reliability Engineer (Systems) - Indeed - Andhra Pradesh   
We are experts in core infrastructure technologies like load balancers, HTTPd, Puppet, Tomcat, Memcached, RabbitMQ, Elasticsearch, MongoDB, and more....
From Indeed - Fri, 23 Jun 2017 07:16:27 GMT - View all Andhra Pradesh jobs
          The Ultimate Data Infrastructure Architect Bundle for $36   
From MongoDB to Apache Flume, This Comprehensive Bundle Will Have You Managing Data Like a Pro In No Time
Expires June 01, 2022 23:59 PST
Buy now and get 94% off

Learning ElasticSearch 5.0


KEY FEATURES

Learn how to use ElasticSearch in combination with the rest of the Elastic Stack to ship, parse, store, and analyze logs! You'll start by getting an understanding of what ElasticSearch is, what it's used for, and why it's important before being introduced to the new features of Elastic Search 5.0.

  • Access 35 lectures & 3 hours of content 24/7
  • Go through each of the fundamental concepts of ElasticSearch such as queries, indices, & aggregation
  • Add more power to your searches using filters, ranges, & more
  • See how ElasticSearch can be used w/ other components like LogStash, Kibana, & Beats
  • Build, test, & run your first LogStash pipeline to analyze Apache web logs

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Ethan Anthony is a San Francisco based Data Scientist who specializes in distributed data centric technologies. He is also the Founder of XResults, where the vision is to harness the power of data to innovate and deliver intuitive customer facing solutions, largely to non-technical professionals. Ethan has over 10 combined years of experience in cloud based technologies such as Amazon webservices and OpenStack, as well as the data centric technologies of Hadoop, Mahout, Spark and ElasticSearch. He began using ElasticSearch in 2011 and has since delivered solutions based on the Elastic Stack to a broad range of clientele. Ethan has also consulted worldwide, speaks fluent Mandarin Chinese and is insanely curious about human cognition, as related to cognitive dissonance.

Apache Spark 2 for Beginners


KEY FEATURES

Apache Spark is one of the most widely-used large-scale data processing engines and runs at extremely high speeds. It's a framework that has tools that are equally useful for app developers and data scientists. This book starts with the fundamentals of Spark 2 and covers the core data processing framework and API, installation, and application development setup.

  • Access 45 lectures & 5.5 hours of content 24/7
  • Learn the Spark programming model through real-world examples
  • Explore Spark SQL programming w/ DataFrames
  • Cover the charting & plotting features of Python in conjunction w/ Spark data processing
  • Discuss Spark's stream processing, machine learning, & graph processing libraries
  • Develop a real-world Spark application

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Rajanarayanan Thottuvaikkatumana, Raj, is a seasoned technologist with more than 23 years of software development experience at various multinational companies. He has lived and worked in India, Singapore, and the USA, and is presently based out of the UK. His experience includes architecting, designing, and developing software applications. He has worked on various technologies including major databases, application development platforms, web technologies, and big data technologies. Since 2000, he has been working mainly in Java related technologies, and does heavy-duty server-side programming in Java and Scala. He has worked on very highly concurrent, highly distributed, and high transaction volume systems. Currently he is building a next generation Hadoop YARN-based data processing platform and an application suite built with Spark using Scala.

Raj holds one master's degree in Mathematics, one master's degree in Computer Information Systems and has many certifications in ITIL and cloud computing to his credit. Raj is the author of Cassandra Design Patterns - Second Edition, published by Packt.

When not working on the assignments his day job demands, Raj is an avid listener to classical music and watches a lot of tennis.

Designing AWS Environments


KEY FEATURES

Amazon Web Services (AWS) provides trusted, cloud-based solutions to help businesses meet all of their needs. Running solutions in the AWS Cloud can help you (or your company) get applications up and running faster while providing the security needed to meet your compliance requirements. This course leaves no stone unturned in getting you up to speed with administering AWS.

  • Access 19 lectures & 2 hours of content 24/7
  • Familiarize yourself w/ the key capabilities to architect & host apps, websites, & services on AWS
  • Explore the available options for virtual instances & demonstrate launching & connecting to them
  • Design & deploy networking & hosting solutions for large deployments
  • Focus on security & important elements of scalability & high availability

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Wayde Gilchrist started moving customers of his IT consulting business into the cloud and away from traditional hosting environments in 2010. In addition to consulting, he delivers AWS training for Fortune 500 companies, government agencies, and international consulting firms. When he is not out visiting customers, he is delivering training virtually from his home in Florida.

Learning MongoDB


KEY FEATURES

Businesses today have access to more data than ever before, and a key challenge is ensuring that data can be easily accessed and used efficiently. MongoDB makes it possible to store and process large sets of data in a ways that drive up business value. Learning MongoDB will give you the flexibility of unstructured storage, combined with robust querying and post processing functionality, making you an asset to enterprise Big Data needs.

  • Access 64 lectures & 40 hours of content 24/7
  • Master data management, queries, post processing, & essential enterprise redundancy requirements
  • Explore advanced data analysis using both MapReduce & the MongoDB aggregation framework
  • Delve into SSL security & programmatic access using various languages
  • Learn about MongoDB's built-in redundancy & scale features, replica sets, & sharding

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Daniel Watrous is a 15-year veteran of designing web-enabled software. His focus on data store technologies spans relational databases, caching systems, and contemporary NoSQL stores. For the last six years, he has designed and deployed enterprise-scale MongoDB solutions in semiconductor manufacturing and information technology companies. He holds a degree in electrical engineering from the University of Utah, focusing on semiconductor physics and optoelectronics. He also completed an MBA from the Northwest Nazarene University. In his current position as senior cloud architect with Hewlett Packard, he focuses on highly scalable cloud-native software systems.

Learning Hadoop 2


KEY FEATURES

Hadoop emerged in response to the proliferation of masses and masses of data collected by organizations, offering a strong solution to store, process, and analyze what has commonly become known as Big Data. It comprises a comprehensive stack of components designed to enable these tasks on a distributed scale, across multiple servers and thousand of machines. In this course, you'll learn Hadoop 2, introducing yourself to the powerful system synonymous with Big Data.

  • Access 19 lectures & 1.5 hours of content 24/7
  • Get an overview of the Hadoop component ecosystem, including HDFS, Sqoop, Flume, YARN, MapReduce, Pig, & Hive
  • Install & configure a Hadoop environment
  • Explore Hue, the graphical user interface of Hadoop
  • Discover HDFS to import & export data, both manually & automatically
  • Run computations using MapReduce & get to grips working w/ Hadoop's scripting language, Pig
  • Siphon data from HDFS into Hive & demonstrate how it can be used to structure & query data sets

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Randal Scott King is the Managing Partner of Brilliant Data, a consulting firm specialized in data analytics. In his 16 years of consulting, Scott has amassed an impressive list of clientele from mid-market leaders to Fortune 500 household names. Scott lives just outside Atlanta, GA, with his children.

ElasticSearch 5.x Cookbook eBook


KEY FEATURES

ElasticSearch is a Lucene-based distributed search server that allows users to index and search unstructured content with petabytes of data. Through this ebook, you'll be guided through comprehensive recipes covering what's new in ElasticSearch 5.x as you create complex queries and analytics. By the end, you'll have an in-depth knowledge of how to implement the ElasticSearch architecture and be able to manage data efficiently and effectively.

  • Access 696 pages of content 24/7
  • Perform index mapping, aggregation, & scripting
  • Explore the modules of Cluster & Node monitoring
  • Understand how to install Kibana to monitor a cluster & extend Kibana for plugins
  • Integrate your Java, Scala, Python, & Big Data apps w/ ElasticSearch

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Alberto Paro is an engineer, project manager, and software developer. He currently works as freelance trainer/consultant on big data technologies and NoSQL solutions. He loves to study emerging solutions and applications mainly related to big data processing, NoSQL, natural language processing, and neural networks. He began programming in BASIC on a Sinclair Spectrum when he was eight years old, and to date, has collected a lot of experience using different operating systems, applications, and programming languages.

In 2000, he graduated in computer science engineering from Politecnico di Milano with a thesis on designing multiuser and multidevice web applications. He assisted professors at the university for about a year. He then came in contact with The Net Planet Company and loved their innovative ideas; he started working on knowledge management solutions and advanced data mining products. In summer 2014, his company was acquired by a big data technologies company, where he worked until the end of 2015 mainly using Scala and Python on state-of-the-art big data software (Spark, Akka, Cassandra, and YARN). In 2013, he started freelancing as a consultant for big data, machine learning, Elasticsearch and other NoSQL products. He has created or helped to develop big data solutions for business intelligence, financial, and banking companies all over the world. A lot of his time is spent teaching how to efficiently use big data solutions (mainly Apache Spark), NoSql datastores (Elasticsearch, HBase, and Accumulo) and related technologies (Scala, Akka, and Playframework). He is often called to present at big data or Scala events. He is an evangelist on Scala and Scala.js (the transcompiler from Scala to JavaScript).

In his spare time, when he is not playing with his children, he likes to work on open source projects. When he was in high school, he started contributing to projects related to the GNOME environment (gtkmm). One of his preferred programming languages is Python, and he wrote one of the first NoSQL backends on Django for MongoDB (Django-MongoDBengine). In 2010, he began using Elasticsearch to provide search capabilities to some Django e-commerce sites and developed PyES (a Pythonic client for Elasticsearch), as well as the initial part of the Elasticsearch MongoDB river. He is the author of Elasticsearch Cookbook as well as a technical reviewer of Elasticsearch Server-Second Edition, Learning Scala Web Development, and the video course, Building a Search Server with Elasticsearch, all of which are published by Packt Publishing.

Fast Data Processing with Spark 2 eBook


KEY FEATURES

Compared to Hadoop, Spark is a significantly more simple way to process Big Data at speed. It is increasing in popularity with data analysts and engineers everywhere, and in this course you'll learn how to use Spark with minimum fuss. Starting with the fundamentals, this ebook will help you take your Big Data analytical skills to the next level.

  • Access 274 pages of content 24/7
  • Get to grips w/ some simple APIs before investigating machine learning & graph processing
  • Learn how to use the Spark shell
  • Load data & build & run your own Spark applications
  • Discover how to manipulate RDD
  • Understand useful machine learning algorithms w/ the help of Spark MLlib & R

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Krishna Sankar is a Senior Specialist—AI Data Scientist with Volvo Cars focusing on Autonomous Vehicles. His earlier stints include Chief Data Scientist at http://cadenttech.tv/, Principal Architect/Data Scientist at Tata America Intl. Corp., Director of Data Science at a bioinformatics startup, and as a Distinguished Engineer at Cisco. He has been speaking at various conferences including ML tutorials at Strata SJC and London 2016, Spark Summit, Strata-Spark Camp, OSCON, PyCon, and PyData, writes about Robots Rules of Order, Big Data Analytics—Best of the Worst, predicting NFL, Spark, Data Science, Machine Learning, Social Media Analysis as well as has been a guest lecturer at the Naval Postgraduate School. His occasional blogs can be found at https://doubleclix.wordpress.com/. His other passion is flying drones (working towards Drone Pilot License (FAA UAS Pilot) and Lego Robotics—you will find him at the St.Louis FLL World Competition as Robots Design Judge.

MongoDB Cookbook: Second Edition eBook


KEY FEATURES

MongoDB is a high-performance, feature-rich, NoSQL database that forms the backbone of the systems that power many organizations. Packed with easy-to-use features that have become essential for a variety of software professionals, MongoDB is a vital technology to learn for any aspiring data scientist or systems engineer. This cookbook contains many solutions to the everyday challenges of MongoDB, as well as guidance on effective techniques to extend your skills and capabilities.

  • Access 274 pages of content 24/7
  • Initialize the server in three different modes w/ various configurations
  • Get introduced to programming language drivers in Java & Python
  • Learn advanced query operations, monitoring, & backup using MMS
  • Find recipes on cloud deployment, including how to work w/ Docker containers along MongoDB

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Amol Nayak is a MongoDB certified developer and has been working as a developer for over 8 years. He is currently employed with a leading financial data provider, working on cutting-edge technologies. He has used MongoDB as a database for various systems at his current and previous workplaces to support enormous data volumes. He is an open source enthusiast and supports it by contributing to open source frameworks and promoting them. He has made contributions to the Spring Integration project, and his contributions are the adapters for JPA, XQuery, MongoDB, Push notifications to mobile devices, and Amazon Web Services (AWS). He has also made some contributions to the Spring Data MongoDB project. Apart from technology, he is passionate about motor sports and is a race official at Buddh International Circuit, India, for various motor sports events. Earlier, he was the author of Instant MongoDB, Packt Publishing.

Cyrus Dasadia always liked tinkering with open source projects since 1996. He has been working as a Linux system administrator and part-time programmer for over a decade. He works at InMobi, where he loves designing tools and platforms. His love for MongoDB started in 2013, when he was amazed by its ease of use and stability. Since then, almost all of his projects are written with MongoDB as the primary backend. Cyrus is also the creator of an open source alert management system called CitoEngine. He likes spending his spare time trying to reverse engineer software, playing computer games, or increasing his silliness quotient by watching reruns of Monty Python.

Learning Apache Kafka: Second Edition eBook


KEY FEATURES

Apache Kafka is simple describe at a high level bust has an immense amount of technical detail when you dig deeper. This step-by-step, practical guide will help you take advantage of the power of Kafka to handle hundreds of megabytes of messages per second from multiple clients.

  • Access 120 pages of content 24/7
  • Set up Kafka clusters
  • Understand basic blocks like producer, broker, & consumer blocks
  • Explore additional settings & configuration changes to achieve more complex goals
  • Learn how Kafka is designed internally & what configurations make it most effective
  • Discover how Kafka works w/ other tools like Hadoop, Storm, & more

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Nishant Garg has over 14 years of software architecture and development experience in various technologies, such as Java Enterprise Edition, SOA, Spring, Hadoop, Hive, Flume, Sqoop, Oozie, Spark, Shark, YARN, Impala, Kafka, Storm, Solr/Lucene, NoSQL databases (such as HBase, Cassandra, and MongoDB), and MPP databases (such as GreenPlum).

He received his MS in software systems from the Birla Institute of Technology and Science, Pilani, India, and is currently working as a technical architect for the Big Data R&D Group with Impetus Infotech Pvt. Ltd. Previously, Nishant has enjoyed working with some of the most recognizable names in IT services and financial industries, employing full software life cycle methodologies such as Agile and SCRUM.

Nishant has also undertaken many speaking engagements on big data technologies and is also the author of HBase Essestials, Packt Publishing.

Apache Flume: Distributed Log Collection for Hadoop: Second Edition eBook


KEY FEATURES

Apache Flume is a distributed, reliable, and available service used to efficiently collect, aggregate, and move large amounts of log data. It's used to stream logs from application servers to HDFS for ad hoc analysis. This ebook start with an architectural overview of Flume and its logical components, and pulls everything together into a real-world, end-to-end use case encompassing simple and advanced features.

  • Access 178 pages of content 24/7
  • Explore channels, sinks, & sink processors
  • Learn about sources & channels
  • Construct a series of Flume agents to dynamically transport your stream data & logs from your systems into Hadoop

PRODUCT SPECS

Details & Requirements

  • Length of time users can access this course: lifetime
  • Access options: web streaming, mobile streaming
  • Certification of completion not included
  • Redemption deadline: redeem your code within 30 days of purchase
  • Experience level required: all levels

Compatibility

  • Internet required

THE EXPERT

Steve Hoffman has 32 years of experience in software development, ranging from embedded software development to the design and implementation of large-scale, service-oriented, object-oriented systems. For the last 5 years, he has focused on infrastructure as code, including automated Hadoop and HBase implementations and data ingestion using Apache Flume. Steve holds a BS in computer engineering from the University of Illinois at Urbana-Champaign and an MS in computer science from DePaul University. He is currently a senior principal engineer at Orbitz Worldwide (http://orbitz.com/).

          Daisuke Konishi: WordPressにElasticsearchを連携してみた話   
いつも使ってる WordPress に検索サービスの Elasticsearch を連携してみた話。 Elasticsearchってどんなものなのか。連携することでどんなことが出来る/出来そうなのかについて話します。

Presentation Slides »


          Java Posse #435   

Roundup ‘13 - Cloud Performance Monitoring

Fully formatted shownotes can always be found at http://javaposse.com

Please join us for the 2014 Java Posse Roundup, from Feb 24th to 28th in Crested Butte, CO. The subject this year is Software Engineering Trends. More details and registration at http://www.mindviewinc.com/Conferences/JavaPosseRoundup/

Thanks

    • Opening - "Java" the parody song Copyright 1997 Broken Records and Marjorie Music Publ. (BMI),

    • Closing - Juan Carlos Jimenez - In the House (Intro No. 1)

  • To contact us:


The Java Posse consists of Tor Norbye, Carl Quinn, Chet Haase and Dick Wall


          Java Posse #434   

Roundup ‘13 - Efficient Searching

Fully formatted shownotes can always be found at http://javaposse.com

Please join us for the 2014 Java Posse Roundup, from Feb 24th to 28th in Crested Butte, CO. The subject this year is Software Engineering Trends. More details and registration at http://www.mindviewinc.com/Conferences/JavaPosseRoundup/

Thanks

    • Opening - "Java" the parody song Copyright 1997 Broken Records and Marjorie Music Publ. (BMI),

    • Closing - Juan Carlos Jimenez - In the House (Intro No. 1)

  • To contact us:


The Java Posse consists of Tor Norbye, Carl Quinn, Chet Haase and Dick Wall


          (USA-MD-Annapolis Junction) Cyber Software Java Engineer   
**Cyber Software Java Engineer** **Requisition ID: 17014899** **Location\(s\): United States\-Maryland\-Annapolis Junction** **US Citizenship Required for this Position: Yes** **Relocation Assistance: No relocation assistance available** **Travel: No** How do cyber terrorists get past the industries best? They don’t\. There are too many of us fighting virtual threats, protecting enterprises and entire countries from large\-scale attacks\. From creating a citywide wireless network for our first responders, to protecting our nation from cyber threats, to building software\-defined radios that change how our military communications, our Information Systems team helps life run smoothly and safely\. If you are the sort of person who loves a challenge and likes to be involved in serious organizational and software change \-\- Then what is happening in the Cyber and Intelligence Mission Solutions Division is the place for you\. This is without a doubt one of the most interesting and exciting times to join an organization like ours\. The culture is one of excellence; team work, learning, delivered value and people\. We are looking for people who love to learn and take initiative to really make this happen\. Northrop Grumman Mission Systems is seeking a Cyber Software Java Engineer to join our team of qualified, diverse individuals located at Annapolis Junction, MD\. Roles and Responsibilities: Work with team lead to understand tasking and deliver system to required specifications\. Work with stakeholders to understand and implement system components\. Participate as a member of the SCRUM team in daily stand ups\. Specific duties include: + Interface with sub\-system software and systems engineers; responsible for gathering user requirements, assisting with implementation, and support to system integration & test\. + Programming to include new development, bug fixes, and enhancements, as well as installation, monitoringand configuration programs and scripts\. + Software testing, installation, configuration and upgrading\. + Perform coding and unit test of new system components as they are defined\. + Documentation of all software developed; documentation to include application and interface documentation, Version Description Documents \(VDD\), User’s Manuals, and Trouble Shooting guides\. Basic Qualifications: + Minimum of 7 years in programs and contracts of similar scope, type, and complexity as a Software Engineer supporting software architecture development, requirements analysis, process execution and evaluation\. + C experience \(5 years\)\. + Experience with streaming frameworks \(IBM Streams, Apache Spark, Storm, etc\.\)\. + Understanding of network & time protocols\. + Ruby, Python, and scripting languages\. + Familiarity with Data Transport formats: JSON, XML\. + Familiarity with ELK stack \(ElasticSearch, LogStash, Kibana\)\. + Linux Experience\. + Version Control Software Experience \(GIT\)\. Preferred Qualifications: + Experience with the Atlassian Suite software development tools\. + Experience in some applicable programming language such as the following: 99. C 99. JAVA 99. Ruby/Python + Experience with No\-SQL and SQL databases\. + Experience integrating and testing components, including unit tests, integration tests, and continuous integration systems\. + Exposure to Agile and SCRUM methodology a plus\. Education: + Bachelor’s Degree in Computer Science or related discipline from an accredited college or university is required\. Four \(4\) years of additional software engineering experience on projects with similar software processes may be substituted for a bachelor’s degree\. Security Clearance:An active TS/SCI Clearance with polygraph is Required\. Northrop Grumman is committed to hiring and retaining a diverse workforce\. We are proud to be an Equal Opportunity/Affirmative Action Employer, making decisions without regard to race, color, religion, creed, sex, sexual orientation, gender identity, marital status, national origin, age, veteran status, disability, or any other protected class\. For our complete EEO/AA and Pay Transparency statement, please visit www\.northropgrumman\.com/EEO \. U\.S\. Citizenship is required for most positions\. **Title:** _Cyber Software Java Engineer_ **Location:** _Maryland\-Annapolis Junction_ **Requisition ID:** _17014899_
          (USA-MD-Annapolis Junction) Senior UI Software Engineer 4   
**Senior UI Software Engineer 4** **Requisition ID: 17014360** **Location\(s\): United States\-Maryland\-Annapolis Junction** **US Citizenship Required for this Position: Yes** **Relocation Assistance: No relocation assistance available** **Travel: No** How do cyber terrorists get past the industries best? They don’t\. There are too many of us fighting virtual threats, protecting enterprises and entire countries from large\-scale attacks\. From creating a citywide wireless network for our first responders, to protecting our nation from cyber threats, to building software\-defined radios that change how our military communications, our Information Systems team helps life run smoothly and safely\. If you are the sort of person who loves a challenge and likes to be involved in serious organizational and software change \-\- Then what is happening in the Cyber and Intelligence Mission Solutions Division is the place for you\. This is without a doubt one of the most interesting and exciting times to join an organization like ours\. The culture is one of excellence; team work, learning, delivered value and people\. We are looking for people who love to learn and take initiative to really make this happen\. Northrop Grumman Mission Systems is seeking a Senior UI Software Engineer 4 to join our team of qualified, diverse individuals located at Annapolis Junction, MD\. Roles and Responsibilities: Software engineer will be responsible for developing front end UI features for a research and development task\. The individual that is selected must have experience with geographic data applications and Mission Web Space Applications\. Basic Qualifications: + 11 years of over IT experience\. + JavaScript\. + JavaScript packages such as leaflet and slickgrid\. + Ability to write nodejs backend web services to access ElasticSearch, MongoDB and Accumulo\. Preferred Qualifications: + Experience with ARCGis\. + RESTful services\. + Linux/Docker\. Education: + Bachelor’s Degree in Computer Science or related discipline from an accredited college or university is required\. Four \(4\) years of additional software engineering experience on projects with similar software processes may be substituted for a bachelor’s degree\. Security Clearance:An active TS/SCI clearance with polygraph is Required\. Northrop Grumman is committed to hiring and retaining a diverse workforce\. We are proud to be an Equal Opportunity/Affirmative Action Employer, making decisions without regard to race, color, religion, creed, sex, sexual orientation, gender identity, marital status, national origin, age, veteran status, disability, or any other protected class\. For our complete EEO/AA and Pay Transparency statement, please visit www\.northropgrumman\.com/EEO \. U\.S\. Citizenship is required for most positions\. **Title:** _Senior UI Software Engineer 4_ **Location:** _Maryland\-Annapolis Junction_ **Requisition ID:** _17014360_
          Firmware Forensics: Diffs, Timelines, ELFs and Backdoors   
This post covers some common techniques that I use to analyze and reverse firmware images. These techniques are particularly useful to dissect malicious firmwares, spot backdoors and detect unwanted modifications.

Backdooring and re-flashing firmware images is becoming mainstream: malicious guys are infecting embedded devices and inserting trojans in order to achieve persistence. Recent articles covered the increasing number of trojanized android firmwares and routers that are being permanently modified.

Attackers with a privileged network position may MITM your requests and forge fake updates containing malicious firmwares. Writing Evilgrade modules for this is really simple, as most vendors keep failing to deliver updates securely, right ASUS?

All your HTTP packets are belong to us...
Older versions of ASUS firmwares were vulnerable to MITM attacks (CVE-2014-2718) because it transmitted updates over HTTP and there were no security/signature checks. ASUS silently patched the issue on 3.0.0.4.376+ and they're now verifying RSA signatures via /sbin/rsasign_check.:

Valid signature -> nvram_set("rsasign_check", "1")

NoConName 2014 CTF Finals: Vodka

I'll keep my tradition of writing posts based on CTF challenges because everybody upvotes CTF posts on reddit it's cool.

The challenge "Vodka", from NoConName 2014 CTF Finals was created by @MarioVilas, who kindly provided the files here (thanks dude!).

I did not participate on the CTF finals, but I found the challenge really interesting because there were many different ways to solve it, summarizing the actions needed to audit a compromised firmware. In my opinion, the best CTF challenges are the ones that require us to develop/use new techniques and improve existing tools.

NoConName 2014 Finals: Vodka
Challenge Category: Forensics
Description: No hints :( just get the flag.

This challenge description is not very intriguing, so I hired a couple of marketing specialists to design a new logo add some Infosec drama and reformulate it:

A mysterious bug affected one of the core routers at a major Internet service provider in Syria. The failure of this router caused the whole country to suddenly lose all connection to the Internet. The Syrian government recorded a traffic capture right before the crash and hired you to perform a forensic analysis.

Download provided: https://github.com/MarioVilas/write-ups/blob/master/ncn-ctf-2014/Vodka/vodka


Network Forensics

The download provided is a packet capture using the PCAP-NG format. Wireshark is too mainstream, so let's convert the PCAP-NG to PCAP and open it using Network Miner:



Network Miner makes it very easy for us to understand what's going on: there's some sort of file transfer via TFTP and the filename seems to be related to an OpenWRT firmware image.


Firmware structure

We always binwalk all the things but very few people stop to analyze and understand the firmware structure properly. We know that the firmware image was downloaded using TFTP, a common way used by many routers to transfer config files/updates and it is probably based on the OpenWRT project.

So what does binwalk tell us?



The Commom Firmware Environment (CFE) is a firmware interface/bootloader present on Broadcom SOCs. It is analogous to the BIOS on PC platforms and it is responsible for CPU initialization and bootstrap code on embedded processors. The CFE is also referred as PMON and it is generally mapped to mtd0.

The JFFS2/NVRAM filesystem is the non-volatile partition. They store all the configuration parameters, including router settings, passwords and logs.

Bear in mind firmware updates generally do not include the CFE/NVRAM partition. You can access the CFE console using serial and you can also dump them on a live system using DD or via SPI. Let's focus on the firmware sections included on the provided image (openwrt-wrtsl54gs-squasfs.bin):

TRX (Offset 0x20)

The TRX header is just an encapsulation, describing a series of information from the firmware, including the image size, CRC, flags, version information and partition offsets. Binwalk wasn't recognizing the header and the relative offsets properly so I submitted these two pull requests. Creating custom signatures for binwalk is pretty straightforward.


Some firmwares (like the newer ones from ASUS and Netgear) use this TRX structure but don't include a loader: the Linux Kernel and the RootFS may be shifted on this occasion.


If the firmware includes any extra header before the TRX, you have to sum their size with the displayed partition offsets in order to find the real values. Some firmwares for SOHO modems out there won't include it, so these values should be right on most cases. The downloaded OpenWRT image had the following offsets:

  • Loader: 0x20 + 0x1C = 0x3C
  • Kernel:  0x20 + 0x8D8 = 0x8F8
  • RootFS: 0x20 + 0x7E400 = 0x7E420

In this specific case, we have a BinHeader right before the TRX, indicating the board ID, the FW Date and the Hardware Date. The struct is described on cyutils.h:

This extra header appears on a few routers like the WRT54G series: the Web GUI checks for this pattern before actually writing the firmware.



We are particularly interested on the fwdate field (Firmware Date), composed by the hex values 07 02 03. According to addpattern.c, the first byte defines the year, the second one is the month and the third byte refers to the day the firmware was created. The fwdate seems to be 03-February-2007, save that for later, we will need that =)

GZ'd LZMA Loader (Offset 0x3C)

According to OpenWRT Wiki, the boot loader has no concept of filesystems: it assumes that the start of the TRX data section is executable code.

The boot loader boots into an LZMA program which decompresses the kernel into RAM and executes it. It turns out the boot loader does know gzip compression, so we have a gzip-compressed LZMA decompression program at 0x3C.

You can find the source code for this lzma-loader here and here. Note the TEXT_START offset at 0x80001000: we may need to adjust the Loading Address on our Disassembler in order to reverse the compiled loader. Don't forget to decompress it (gunzip) before reversing the file.


Most embedded toolchains would strip the binaries in order to reduce the firmware size. If you want to reverse a friendlier version of the loader, grab the latest OpenWRT ImageBuilder and search for loader.elf:


Woohoo, blue code =)

Note that if we modify the loader to include a backdoor, we would have our very own Router Bootkit, cool isn't it?

LZMA'd Kernel (Offset 0x8F8)

Instead of just putting a kernel directly onto flash, most embedded devices compress the kernel using LZMA. The boot loader boots into an LZMA program which decompresses the kernel into RAM and executes it.

Binwalk has a signature to find Kernel strings in raw Linux Kernels. The identified string lists the toolchain used to compile the Kernel, as well as the compiled date and version information:


And why did binwalk manage to find all these information from the Kernel? The answer can be found on the toolchain's Makefile:

If we follow the steps from my previous post we can build a customized Kernel for OpenWRT. The generated vmlinux is generally an ELF file, but in our case, the object was stripped using objcopy:

Did you notice the compile date was 03-February-2007? Let's save that for later as well.

SquashFS (Offset 0x72420)

The last part is the actual filesystem. Most embedded Linux devices use SquashFS and many vendors hack it in order to get better compression and faster performance. Hopefully we don't have to worry about that as Sasquatch handles different SquashFS header/compression formats.

The filesystem has the standard OpenWRT directories and files, including a banner from the 0.9 build (White Russian).


Both binwalk and sasquatch display the SquashFS superblock information, including the creation/last append time:


Did you spot the date 29-October-2014? There's definitely something going on here =)


Directory Tree Diff & Fuzzy Hashing

Now that we have unpacked & unsquashed the firmware, let's use binwally to compare the directory tree and find the needle in the haystack.

After googling the filename (openwrt-wrtsl54gs-squashfs.bin), we get three possible candidates:

https://downloads.openwrt.org/whiterussian/0.9/default/openwrt-wrtsl54gs-squashfs.bin
https://downloads.openwrt.org/whiterussian/0.9/micro/openwrt-wrtsl54gs-squashfs.bin
https://downloads.openwrt.org/whiterussian/0.9/pptp/openwrt-wrtsl54gs-squashfs.bin

OpenWRT offers different builds for the same device because of constraints like limited flash size. Let's download these three candidates, unpack and compare them:

binwally.py ctf/_openwrt-wrtsl54gs-squashfs.bin.extracted/ micro/_openwrt-wrtsl54gs-squashfs.bin.extracted/


The "micro" build has the highest overall match score (99%), let's spot the differences:

binwally.py ctf/_openwrt-wrtsl54gs-squashfs.bin.extracted/ micro/_openwrt-wrtsl54gs-squashfs.bin.extracted/ | grep -E -v "ignored|matches"


After carefully reviewing these files, we notice that the "/etc/profile" was modified to include a call to the nc backdoor.


The LZMA'd Kernel (offset 0x8F8) is the same on both images, even though binwally reports a difference. This happens because binwalk extraction doesn't know when to stop and both files also contain additional data like the SquashFS partition.

The backdoor located at "/bin/nc" is a simple bash script that checks the MD5 from "/etc/profile" and draws a Nyan Cat along with the challenge key. In order to get the proper key, we simply modify the file location to the relative path "./etc/banner", to avoid overlapping with the file from the original system.


After running the file, we get the key NCNdeadb6adec4c77a40c23e04770924d3c5b18face.


This was just too easy right? But what if we didn't have a known template for comparison?


Timeline Analysis

My tool of choice to perform timeline analysis is Plaso, created by @el_killerdwarf. The tool is python-based, modular and very fast. What I like most about it is the ease to output results to ELK. If you don't know about Plaso and the ELK stack, read this quick tutorial and set up your environment.

Let's use log2timeline to create a dump file, pointing to the extracted SquashFS path:

log2timeline.py output.dump squashfs-root/


Let's fire up psort and include data in the timeline:

psort.py -o elastic output.dump


That's all, Plaso uses the filestat parser to extract metadata from the files, outputting results to Elasticsearch.

We already identified the following dates from the firmware:

  • 03 February 2007 (??:??:??): BinHeader firmware creation date
  • 03 February 2007 (13:16:08): Linux Kernel compile date
  • 29 October  2014 (16:53:25): SquashFS creation or last append time

First let's filter the filesystem attributes: we just want to display the mtime (modified) timestamp, so we are going to perform a micro analysis to include the value. The filter should be something like this: field must | field timestamp_desc | query: "mtime".


The histogram view is very helpful to get a big picture of what's going on:


We can clearly see that the files included/modified on 2014-10-29 had a malicious nature. The state sponsored attacker did not modify other files from the OpenWRT base image.


At this point it is pretty clear that the firmware was modified using the OpenWRT Image Builder, which is a pre-compiled OpenWrt build environment. The BinHeader and the Kernel timestamps were left untouched and the only partition modified was the SquashFS one.

Of course these timestamps, like any kind of metadata, could be tampered by the malicious hacker. However, they are very helpful during the initial phases, speeding up investigations and narrowing the analysis to a smaller set of data.


ELF Structural Information

I always get impressed when AV vendors manage to profile APT and State-sponsored attackers based on PE timestamps. Techniques like the imphash are generally used exclusively on Windows.

PE Imports are the functions that a piece of software calls from other files (typically DLLs). To track these imports, a hash is created based on library/API names and their specific order within the executable. Because of the way a PE’s import table is generated, we can use the imphash value to identify related malware samples, for example.

Everybody does that for Windows binaries but what about Linux? Virustotal recently included detailed ELF information on their engine. We can also use these sections to identify useful information from the binaries, including the toolchain used to compile them.

We generally don't have any timestamp information on the ELF section, but there are many other interesting fields. This quick guide on using strip summarizes some topics:

When an executable is produced from source code, there are two stages - compilation and linking. Compiling takes a source file and produces an object file. Linking concatenates these object files into a single executable. The concatenation occurs by section. For example, the .comment section for the final executable will contain the contents of the .comment section of each object file that was linked into the executable.
If we examine the contents of the .comment section we can see the compiler used, plus the version of the compiler
It's pretty simple to read and parse the .comment sections from ELF files. GNU readelf (part of binutils) and pyelftools include all the necessary functions parse them.

I always try to display information from object files using different toolchains in order to find out which one understands the file structure properly. On this specific case, I'm going to use mipsel-linux-gnu-readelf (part of Emdebian toolchain), but the regular readelf also does the job.

for i in $(find .) ; do echo $i ; mipsel-linux-gnu-readelf -p .comment $i ; done > comment-section.txt
./lib/modules/2.4.30/diag.o

String dump of section '.comment':
  [     1]  GCC: (GNU) 3.4.4 (OpenWrt-1.0)

./lib/modules/2.4.30/switch-adm.o

String dump of section '.comment':
  [     1]  GCC: (GNU) 3.4.4 (OpenWrt-1.0)

./lib/modules/2.4.30/switch-robo.o

String dump of section '.comment':
  [     1]  GCC: (GNU) 3.4.4 (OpenWrt-1.0)

./lib/modules/2.4.30/switch-core.o

String dump of section '.comment':
  [     1]  GCC: (GNU) 3.4.4 (OpenWrt-1.0)

./lib/modules/2.4.30/wlcompat.o

String dump of section '.comment':
  [     1]  GCC: (GNU) 3.4.4 (OpenWrt-1.0)

Just a few ELF files included the comment section, others got stripped during the compilation/linking phase. If we download OpenWRT 0.9 sources we can see that GCC 3.4.4 was indeed used:


TheMoon Worm exploited a command injection to infect Linksys wireless routers with a self-replicating malware. If we analyze its .comment section, we can see that it was probably compiled and linked using GCC 4.2.4 and 3.3.2. If we search for a .comment section on the router E4200, targeted by the worm, we can't find any reference because the toolchain stripped all of them. Having a file compiled with a different toolchain or containing extra ELF sections (that others files don't) is something highly suspicious.


The .comment section for the final executable includes the contents of the .comment section of each object file that was linked into the executable. If we compare the comment section on ASUS RT-AC87U Firmwares v3.0.0.4.378.3885 and v3.0.0.4.376.2769, we can spot an extra line on the newer version from tfat.ko:


If you want to dump all sections from the ELF file you may use this command line (kind of hacky, but works):

for i in $(find .) ; do echo "$i" ; for j in $(readelf -S "$i" | grep \\[ | cut -d"]" -f2 | cut -d " " -f2 | grep -v "Name") ; do mipsel-linux-gnu-readelf -p "$j" "$i" ; done ; done > list.txt

The output will be a bit too verbose, you may want to narrow the analysis to the following sections:

  • .comment - contains version control information
  • .modinfo - displays information from a kernel module
  • .notes - comments put there by the compiler/linker toolchain
  • .debug - contains information for symbol debugging
  • .interp - contains the name of the dynamic loader

For more information regarding the ELF file structure, check the ELF man and the Chapter 5 from Malware Forensics Field Guide for Linux Systems.


Conclusion

Without further clues or context these information may not be relevant, but in conjunction with other data they're helpful to get a big picture of what's going on:

  • Diffing the content from previous firmwares may be useful to find out when backdoors were first installed, modified and/or removed.

  • Artifact timeline creation and analysis also helps to speed up investigations by correlating the vast amount of information found on system.

  • The contents from the ELF section will likely reveal the toolchain and the compiler version used to compile a suspect executable. Clues such as this are attribution identifiers, contributing towards identifying the platform used by the attacker to craft his code.

We can use the timestamps from the kernel partition to correlate different firmwares from the same family, for example. We can also compare the timestamps from each partition to find deviations: a firmware header created on 2007, with a Kernel timestamp from 2007 and a SquashFS partition dated to 2014 is highly suspicious.

The Firmware.RE project is performing a large scale analysis, providing a better understanding of the security issues related to firmwares. A broader view on firmwares is not only beneficial, but necessary to discover new vulnerabilities and backdoors, correlating different device families and showing how vulnerabilities reappear across different products. This is a really cool project to track how firmwares are evolving and getting security fixes.




          Senior Software Developers   
MA-Woburn, Senior Software Developers. Multiple openings. Headquarters: Woburn, MA. Analyze requirements and prepare requirement specifications and technical design documents. Design SQL database clusters for cloud platforms. Design control repositories and Elasticsearch and Hadoop clusters. Automate archival process and develop Kibana dashboards. Review service failures, analyze root cause, and provide solu
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Systems Engineer - Zipwhip, Inc. - Seattle, WA   
Riak, Cassandra, MongoDB, ElasticSearch, etc. We are looking for an Systems Engineer for our DevOps team that will help support and grow our systems and network...
From Indeed - Fri, 12 May 2017 22:07:41 GMT - View all Seattle, WA jobs
          (USA-TX-Austin) DevOps/Linux Architect   
DevOps/Linux Architect DevOps/Linux Architect - Skills Required - Linux, Puppet, Ansible, Docker, Nginx, Thoughtworks Go, MySQL, Python, BASH, Perl Headquartered in Austin, TX we are an online services provider that has developed Internet applications and Security software for over 20 years. Due to rapid growth, we are looking for an outstanding DevOps/Linux Architect to design, improve, and manage the infrastructure for our internal and external environments. We are looking for an Architect who is passionate about using the latest, cutting edge technologies and implementing and managing our IT systems. **What You Will Be Doing** + Be the subject matter expert between DevOps and Linux Systems Engineering teams + Provide architectural perspective and vision as top-level contributor in the Enterprise + Technical lead on projects + Provide top level support for multiple company products + Develop internal tools and processes to keep our infrastructure performance high and stable **What You Need for this Position** MUST HAVE + Experience implementing and managing Linux-based Enterprise tech projects in a 24/7 environment + Experience implementing and managing IT systems + Architect-level knowledge of DevOps technologies (Ansible, Puppet, Docker, NGINX, Thoughtworks Go are preferred) + Scripting (Python, Bash or Perl) + Knowledge of TCP/IP networking and DNS NICE TO HAVE + Splunk and ElasticSearch + MySQL/MariaDB, PostgreSQL **What's In It for You** + Competitive Base Salary + Generous profit-sharing plan & PTO + Comprehensive benefits (medical, dental, vision) + Flexible schedule (Remote 2x a week) So, if you are a DevOps/Linux Architect passionate about building and deploying IT systems, please apply today! Applicants must be authorized to work in the U.S. **CyberCoders, Inc is proud to be an Equal Opportunity Employer** All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status, or any other characteristic protected by law. **Your Right to Work** – In compliance with federal law, all persons hired will be required to verify identity and eligibility to work in the United States and to complete the required employment eligibility verification document form upon hire. *DevOps/Linux Architect* *TX-Austin* *MR3-1382427*
          (USA-MD-Annapolis Junction) Cyber Software Java Engineer   
**Cyber Software Java Engineer** **Requisition ID: 17014899** **Location\(s\): United States\-Maryland\-Annapolis Junction** **US Citizenship Required for this Position: Yes** **Relocation Assistance: No relocation assistance available** **Travel: No** How do cyber terrorists get past the industries best? They don’t\. There are too many of us fighting virtual threats, protecting enterprises and entire countries from large\-scale attacks\. From creating a citywide wireless network for our first responders, to protecting our nation from cyber threats, to building software\-defined radios that change how our military communications, our Information Systems team helps life run smoothly and safely\. If you are the sort of person who loves a challenge and likes to be involved in serious organizational and software change \-\- Then what is happening in the Cyber and Intelligence Mission Solutions Division is the place for you\. This is without a doubt one of the most interesting and exciting times to join an organization like ours\. The culture is one of excellence; team work, learning, delivered value and people\. We are looking for people who love to learn and take initiative to really make this happen\. Northrop Grumman Mission Systems is seeking a Cyber Software Java Engineer to join our team of qualified, diverse individuals located at Annapolis Junction, MD\. Roles and Responsibilities: Work with team lead to understand tasking and deliver system to required specifications\. Work with stakeholders to understand and implement system components\. Participate as a member of the SCRUM team in daily stand ups\. Specific duties include: + Interface with sub\-system software and systems engineers; responsible for gathering user requirements, assisting with implementation, and support to system integration & test\. + Programming to include new development, bug fixes, and enhancements, as well as installation, monitoringand configuration programs and scripts\. + Software testing, installation, configuration and upgrading\. + Perform coding and unit test of new system components as they are defined\. + Documentation of all software developed; documentation to include application and interface documentation, Version Description Documents \(VDD\), User’s Manuals, and Trouble Shooting guides\. Basic Qualifications: + Minimum of 7 years in programs and contracts of similar scope, type, and complexity as a Software Engineer supporting software architecture development, requirements analysis, process execution and evaluation\. + C experience \(5 years\)\. + Experience with streaming frameworks \(IBM Streams, Apache Spark, Storm, etc\.\)\. + Understanding of network & time protocols\. + Ruby, Python, and scripting languages\. + Familiarity with Data Transport formats: JSON, XML\. + Familiarity with ELK stack \(ElasticSearch, LogStash, Kibana\)\. + Linux Experience\. + Version Control Software Experience \(GIT\)\. Preferred Qualifications: + Experience with the Atlassian Suite software development tools\. + Experience in some applicable programming language such as the following: 99. C 99. JAVA 99. Ruby/Python + Experience with No\-SQL and SQL databases\. + Experience integrating and testing components, including unit tests, integration tests, and continuous integration systems\. + Exposure to Agile and SCRUM methodology a plus\. Education: + Bachelor’s Degree in Computer Science or related discipline from an accredited college or university is required\. Four \(4\) years of additional software engineering experience on projects with similar software processes may be substituted for a bachelor’s degree\. Security Clearance:An active TS/SCI Clearance with polygraph is Required\. Northrop Grumman is committed to hiring and retaining a diverse workforce\. We are proud to be an Equal Opportunity/Affirmative Action Employer, making decisions without regard to race, color, religion, creed, sex, sexual orientation, gender identity, marital status, national origin, age, veteran status, disability, or any other protected class\. For our complete EEO/AA and Pay Transparency statement, please visit www\.northropgrumman\.com/EEO \. U\.S\. Citizenship is required for most positions\. **Title:** _Cyber Software Java Engineer_ **Location:** _Maryland\-Annapolis Junction_ **Requisition ID:** _17014899_
          (USA-MD-Annapolis Junction) Senior UI Software Engineer 4   
**Senior UI Software Engineer 4** **Requisition ID: 17014360** **Location\(s\): United States\-Maryland\-Annapolis Junction** **US Citizenship Required for this Position: Yes** **Relocation Assistance: No relocation assistance available** **Travel: No** How do cyber terrorists get past the industries best? They don’t\. There are too many of us fighting virtual threats, protecting enterprises and entire countries from large\-scale attacks\. From creating a citywide wireless network for our first responders, to protecting our nation from cyber threats, to building software\-defined radios that change how our military communications, our Information Systems team helps life run smoothly and safely\. If you are the sort of person who loves a challenge and likes to be involved in serious organizational and software change \-\- Then what is happening in the Cyber and Intelligence Mission Solutions Division is the place for you\. This is without a doubt one of the most interesting and exciting times to join an organization like ours\. The culture is one of excellence; team work, learning, delivered value and people\. We are looking for people who love to learn and take initiative to really make this happen\. Northrop Grumman Mission Systems is seeking a Senior UI Software Engineer 4 to join our team of qualified, diverse individuals located at Annapolis Junction, MD\. Roles and Responsibilities: Software engineer will be responsible for developing front end UI features for a research and development task\. The individual that is selected must have experience with geographic data applications and Mission Web Space Applications\. Basic Qualifications: + 11 years of over IT experience\. + JavaScript\. + JavaScript packages such as leaflet and slickgrid\. + Ability to write nodejs backend web services to access ElasticSearch, MongoDB and Accumulo\. Preferred Qualifications: + Experience with ARCGis\. + RESTful services\. + Linux/Docker\. Education: + Bachelor’s Degree in Computer Science or related discipline from an accredited college or university is required\. Four \(4\) years of additional software engineering experience on projects with similar software processes may be substituted for a bachelor’s degree\. Security Clearance:An active TS/SCI clearance with polygraph is Required\. Northrop Grumman is committed to hiring and retaining a diverse workforce\. We are proud to be an Equal Opportunity/Affirmative Action Employer, making decisions without regard to race, color, religion, creed, sex, sexual orientation, gender identity, marital status, national origin, age, veteran status, disability, or any other protected class\. For our complete EEO/AA and Pay Transparency statement, please visit www\.northropgrumman\.com/EEO \. U\.S\. Citizenship is required for most positions\. **Title:** _Senior UI Software Engineer 4_ **Location:** _Maryland\-Annapolis Junction_ **Requisition ID:** _17014360_
          Site Reliability Engineer (Systems) - Indeed - Andhra Pradesh   
We are experts in core infrastructure technologies like load balancers, HTTPd, Puppet, Tomcat, Memcached, RabbitMQ, Elasticsearch, MongoDB, and more....
From Indeed - Fri, 23 Jun 2017 07:16:27 GMT - View all Andhra Pradesh jobs
          Senior Full Stack Web Developer - (Boxborough)   
Location Boxborough Massachusetts US Area of Interest Engineer Software Job Type Professional Technology Interest Collaboration Video Job Id New What You ll Do Research technical approaches that address full stack requirements Implement new features and extend existing functionality Connect business requirements with technical solutions and vice versa Use static analysis code complexity and other state of the art softwaredevelopment tools and processes to develop high quality software Develop efficient and effective automated unit and feature tests for new andexisting software Who You ll Work With We are looking for an experienced engineer that understands and has implementedfull stack web development for customer cloud based products solutions Yourjob will be to implement our next generation customer communication andcollaboration solution that is user experience focused This will involvecollaborating with other engineering teams at Cisco determining the righttechnical approach to achieve the system s goals and contributing to thedevelopment of the solution The ideal candidate will have the drive and desireto solve real problems for customers the architectural perspective to make theright technical decisions and the entrepreneurial spirit to personally codekey components Who You Are years of software engineering experience Experienced a developing highly available and reliable cloud based applications Has successfully implemented Web APIs and UserInterfaces using REST Java JavaScript REACT and Node JS Understanding of instrumenting code for cloud deployments Understanding TDD and Continuous Integration andContinuous Delivery Experience with some of the following Cassandra ElasticSearch and or Redis Experienced developing services on AWS OpenStack Cloud Foundry Heroku orsimilar Understanding of Security technology and challenges a plus Experience working on an Agile Scrum team project Why Cisco We connect everything people processes data and things We innovateeverywhere taking bold risks to shape the technologies that give us smartcities connected cars and handheld hospitals And we do it in style withunique personalities who arena t afraid to change the way the world works lives plays and learns We are thought leaders tech geeks pop culture aficionados and we even have afew purple haired rock stars We celebrate the creativity and diversity thatfuels our innovation We are dreamers and we are doers dicedev Source: http://www.juju.com/jad/000000009m39bg?partnerid=af0e5911314cbc501beebaca7889739d&exported=True&hosted_timestamp=0042a345f27ac5dc69354e46c76daa485f5433b1779459d32f1da485eef8e872
          ElasticSearch 5.4.3 发布,分布式搜索引擎   

ElasticSearch 5.4.3 发布了。ElasticSearch 是一个基于 Lucene 构建的开源,分布式,RESTful 搜索引擎。设计用于云计算中,能够达到实时搜索,稳定,可靠,快速,安装使用方便。支持通过 HTTP 使用 JSON 进行数据索引。

更新内容:

Enhancements

  • Snapshot/Restore

    • Improves snapshot logging and snapshot deletion error handling #25264

Bug fixes

  • Snapshot/Restore

    • Remove redundant and broken MD5 checksum from repository-s3 #25270 (issue: #25269)

下载地址:


          Python分布式爬虫打造搜索引擎完整版-基于Scrapy、Redis、elasticsearch和django打造一个完整的搜索引擎网站   

          Elasticsearch cloud partnership plays to Google's base   
Google's cloud partnership with Elastic brings the open source search engine and analytics tool to GCP, as Google tries to appeal to enterprises without ignoring its customer base.
          (IT) Platform Architect - AWS/Azure   

Rate: £600 - £680 per Day   Location: Redhill, Surrey   

Platform Architect - AWS/Azure As the Platform Architect, you will be working with teams of engineers, in a hands-on capacity, to build required platform components. You will lead, design and implement decisions relating to the new application platform infrastructure being built and expanded for the hosting of our digital applications. Essential Requirements: -Very strong AWS experience, ideally with certifications -Solid knowledge of networking (particularly in AWS), databases and web Servers -We require you to have and a deep understanding of micro services, distributed computing architectures and associated platforms to make right tools and technology choices -Experience of implementing micro services architectures and associated non-functional requirements (including Scalability, Security, Load Balancing, Monitoring and Logging) -Experience hosting high profile public websites and associated technologies (CDNs, WAFs etc) -Experience of Docker based platforms and container orchestration/scheduling, ideally Amazon Beanstalk and ECS -Strong hands on skills as a platform engineer including Shell Scripting and either ruby or python The Role will be responsible for the following areas: -This is a highly influential role that requires a mix of technical acumen and technical leadership skills to help drive the Digital platform ecosystem. This person must be creative, ask questions, and be comfortable challenging the status quo. -Working alongside delivery teams to ensure design decisions are understood, tested and implemented. -Looking ahead to identify next of changes required to the platform to meet requirements based on predicted growth and complexity and maintain a rolling roadmap Desired requirements: -Software engineering background, ideally Java -Experience in any of the following technologies; Docker, AWS Beanstalk, Puppet, Travis, Elasticsearch, Dynamo DB -Experience of DevOps culture, engineering practices and Kanban driven delivery Rate: Up to £680 per day Duration: Initial 6 months Location: Surrey Interquest Group PLC is acting as an Employment Business in relation to this vacancy.
 
Rate: £600 - £680 per Day
Type: Contract
Location: Redhill, Surrey
Country: UK
Contact: Louis Fitch
Advertiser: InterQuest Financial Markets
Start Date: ASAP
Reference: JS-LDFPA

          Product Owner/Business Analyst - SemanticBits - Remote   
You will be part of a highly talented, dynamic development team that will be using a set of modern technologies (AngularJS, Node.js, RoR, MongoDB, ElasticSearch...
From SemanticBits - Tue, 02 May 2017 23:21:26 GMT - View all Remote jobs
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Software Developer 2 - Lawrence Berkeley National Laboratory - Berkeley, CA   
Experience with SQL (such as MySQL and postgres) or NoSQL databases (such as MongoDB, Lucene, SOLR or ElasticSearch)....
From Lawrence Berkeley National Laboratory - Tue, 13 Jun 2017 22:58:55 GMT - View all Berkeley, CA jobs
          Systems Engineer - Zipwhip, Inc. - Seattle, WA   
Riak, Cassandra, MongoDB, ElasticSearch, etc. We are looking for an Systems Engineer for our DevOps team that will help support and grow our systems and network...
From Indeed - Fri, 12 May 2017 22:07:41 GMT - View all Seattle, WA jobs
          michaeljennings/laralastica (v2.1.7)   
A laravel package that allows you to search eloquent results using elasticsearch.
          Kibana 5.X で visualize 結果から elasticsearch query を取得したい   

kibana の visualize 機能で集計グラフを作成できたけど、これを Elasticsearch から直接取得したい。 Query に直さなきゃいけないんだけど、さすがに手でやるのは面倒だし…

Visualize 画面下部の矢印を押すと、spy mode になる。

  • Table (検索結果の Table view)
  • Request Body
  • Response Body
  • Statistics (クエリのレスポンスタイムなど)

これらを表示できるので、Request Body を見ればよい。

書き終わった後に気づいたよ!!

出典: github.com


          Site Reliability Engineer (Systems) - Indeed - Andhra Pradesh   
We are experts in core infrastructure technologies like load balancers, HTTPd, Puppet, Tomcat, Memcached, RabbitMQ, Elasticsearch, MongoDB, and more....
From Indeed - Fri, 23 Jun 2017 07:16:27 GMT - View all Andhra Pradesh jobs