Apache Flink is a platform for distributed stream data processing. It is an efficient tool for manipulation with large amounts of data.

Flink was invented in Berlin Techical University (“Flink” means “quick” in German). It is a primary competitor to Apache Spark. The point is that Apache Flink works as a pure streaming engine, when Spark Streaming framework is just a fast micro-batch processing. Otherwise, Flink allows batch mode as a special case of processing.

Of course, Apache Flink has an API for Java and Scala. Inside the core it provides DataSet and DataStream APIs for working with batch and streams. There’s also a distributed runtime for dataflows.

Apache Flink libraries

Apache Flink has a stack of high-level APIs for Machine Learning (Flink ML), Graphs (Gelly) and Relational Data (Table API) processing. At this moment these libs are in beta-status.

Flink could be integrated with a bunch of well-known Big Data tools. You must know that it runs on YARN and works on top of Hadoop Distributed File System (HDFS). It can perform data streaming from Kafka, HBase and many other datasources.

Apache Flink Basics – sample code

Now let’s see the code for the simplest task to represent Flink basics with the DataSet API. We will count the number of “Flink” occurrences in a sample text.

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

DataSet<String> lines = env.fromElements("This is a test data for Apache Flink",
        "In the end Flink will print the number",
        "of Flink occurrences in this text.",
        "Good luck!");
DataSet<Integer> flinkCounts = lines
        .flatMap(new FlatMapFunction<String, Integer>() {
            @Override
            public void flatMap(String line, Collector<Integer> collector) throws Exception {
                  if (line.contains("Flink")) collector.collect(1);
            }
        }).reduce(new ReduceFunction<Integer>() {
            @Override
            public Integer reduce(Integer first, Integer second) throws Exception {
                  return first + second;
            }
        });

flinkCounts.print();

We have a working example of Word Count task with Flink, you can see it here.

There’s also an article with Flink vs Spark comparison.