One of the most famous framework to support map-reduce for large scale data processing, a.k.a. BigData, is Hadoop as I introduced almost two years ago here. Data processing wise, the Java 8 stream API can do pretty much the same. Here is a quick sample that shows how it count number of words in string. There are significant differences in how they are implemented and the cases in which they should be used. Let’s discuss them after the sample.
Java Collections APIs consists of well designed classes and interfaces for managing all sorts of data structures. With Java 8, there is a new enhancement called Stream API related to the Collection APIs (see What’s New in JDK 8). I spent some time to study the new feature last week as part of my effort to bring myself up to date with Java 8.
Coming with lambda feature in Java 8 is the method reference feature. It allows programmers to use methods as variables, and parameters to other methods. It works for class constructors too, where the method name is unanimously “new.’
There are four types of method references per Oracle documentation. Syntax wise, they all look similar with double colons as shown in the following sample code.
Following the last article about Lambda feature in Java 8, I spent some more time on other features and will blog about them along the way. It’s a good way to push me to really think through and thus deepen my understanding. This article is about default method which is in fact related to the lambda feature.
On last Tuesday (March 18, 2014), Oracle released the Java SE 8. Here is the official documentation from the company on the new features in Java SE 8, ranging from language itself, libraries, tool, and run-time.
In my last article, I discussed on the format requirement for Java APIs and how I found out the root cause and its solution. Even more mysterious is the format requirement of VMware vCenter as I discovered in another VMware related project, in which I needed to register an extension with vCenter and set up its certificate.
I got into a very interesting problem while writing code to read a X.509 certificate. It’s a standard PEM encoded certificate (shown below) as you would find anywhere else.
The exception is as follows. It’s pretty clear, but also confusing because the certificate string has included both BEGIN and END. It seems to me very complete. Of course, the program does not lie and we have to trust it most of the time unless there is a bug.
As I introduced in last article, there are two sets of APIs in UCS Director: north bound APIs, and south bound APIs. The north bound APIs are REST styled, allowing other applications to invoke UCS Director functionalities, or simply retrieve information from UCS Director. We’ll go through the REST APIs in details so that you can quickly get started with it.
If you’ve followed my blog, you’ve read several articles on Nginx. This is the last article of the series, and also an interesting one. While troubleshooting a system I helped to build, I found a very challenging problem that I think is worthwhile to share. The system consists of a client and a server, between which is a Nginx server as reversed proxy. This is a typical setting for flexibility and control. Somehow, the client got disconnected pretty consistently but the back end work went through successfully. It’s really a false alarm but not good for a product.
I just went through a two day training course on Cisco UCS Director APIs that covers both the REST APIs and Open Automation SDK. For people who don’t know UCS Director yet, it’s the orchestration engine Cisco acquired from a start-up company Cloupia more than one year ago. If you know VMware vCenter Orchestrator, UCS Director is something very similar but with more features on various hardware components for converged infrastructure. To fit into its unified data center strategy, Cisco re-branded it as UCS Director.
In a recent project using Nginx as reversed proxy server, I got into an interesting problem: how can the server behind the Nginx tell whether a HTTP request comes from remote host or local host? If I just look at the IP header of the request, they are all local because the reversed-proxied packets from remote seems to be sent from local as well. So I cannot really tell the source of a HTTP request.
While debugging a vSphere Web Client plugin project, I found it’s not easy to refresh the services with the Virgo server which acts as the back end for the plugin GUI but as client for the vCenter server. Packaged as OSGi bundle, it’s supposed to be easy to reload the service. Mixed together with various components in the plugins, however, it’s sometimes not quite straight forward for the re-deployment for updated code. Here is a brute force approach I found while playing with it.
In some use cases, you want to protect different parts of a Web application with different approaches. For example, the admin related resources normally require stronger mechanism than the user related ones. The following I will show how to use Nginx with client side certificate for the resources under /admin namespace for admins, and user/name for normal users.
Generating Certificates and Keys
Base64 is a straight forward encoding for binary data into readable characters (RFC 4648 and RFC 2045). Although you can do it by yourself, more often than not you would like use an existing library, for example, Apache common. If you just need the Base64 encoding but have to import the whole library, it’s not a good idea. There is actually a better way without introducing extra dependency, which is to hack Java standard library 1.6.
As I develop software, I rarely need to manage several servers using SSH at same time. If I do, I just manually connect to each server and type same commands over and over. Of course, it takes much time for the repeated work. More importantly, it’s very hard to repeat the steps consistently across multiple servers especially when there are more than 4 servers.
We’ve covered Apache Thrift in last few articles from simple HelloWorld sample, Python Thrift client, to the securing Thrift traffic. Here I am going to discuss more on user authentication, which is a must for protecting the services and user authorization. This is in general a weakness of Thrift, but could be solved with different approaches. Having said that, if you have chosen Thrift, you probably build internal system where user access control is not important.
Thrift is mostly used for distributed systems which run mostly in house. There is no strong demand for securing the traffic on the wire. There are however use cases in which the Thrift services are exposed as a public service. In these use cases, the Thrift traffic should be secured with SSL/TLS. It comes with a price which more work on client and server on encryption and decryption. This is not a big deal for light load server, but for heavy load server it could be a problem. It can be mitigated with hardware acceleration on load balance servers between which and the client can be SSL, but not after that to the Thrift server.
As mentioned in my last post, Thrift is a cross-language and cross platform RPC framework. We’ve seen how a Java based Thrift server and client work there. Let’s take a look how to write a quick python script that connects to the Java Thrift server. It’s all possibe to write a Python based Thrift server, but probably not what most people want to do due to performance and scalability.
Thrift is one of the RPC frameworks that are widely used nowadays. It’s originally developed at Facebook and then open sourced under Apache Foundataion. It’s supported by major programming or scripting languages like Java, C++, Python, Ruby, etc. The typical use case is for building distributed systems, mostly in house.
As discussed in my last post, after installing and configuring Nginx as the reversed proxy server for Tomcat, it’s necessary to block remote access to the original port served by Tomcat. To achieve this, iptables should be a good solution. Simpler solution is to change one line in the Tomcat server configuration file so that Tomcat accepts only requests from local host.
With Tomcat 7 on Ubuntu, the configuration file is /var/lib/tomcat7/conf/server.xml. Just add address=”127.0.0.1″ into the related Connector section as follows: