SCALED VOID

[DRAFT] Debugging FLINK-36808 Issue

2025-11-20T00:00:00+00:00

Recently I picked up the FLINK-36808 bug in the Apache Flink project. The solution to bug covers topics such as SQL LookupJoin, Flink SQL Planner, and Volcano optimizer (which is provided by the Apache Calcite project).

In this post I describe my experience debugging the issue and the lessons learned along the way. You can already find the pull request with the fix here PR #26514.

The issue was internally reported by my colleague Jun Qin and initial investigations were done by my teammate Qingsheng Ren. Let us start by first trying to understand the issue.

As mentioned in the ticket, given the union query of two lookup joins on the dimension tables, we get wrong results.

-- Data of table `stream`:
-- (1, Alice)
-- (2, Bob)
CREATE TEMPORARY TABLE `stream` (
    `id` BIGINT,
    `name` STRING,
    `txn_time` as proctime(),
    PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:postgresql://localhost:5432/postgres',
  'table-name' = 'stream',
  'username' = 'postgres',
  'password' = 'postgres'
);

-- Data of table `dim`:
-- (1, OK)
-- (2, OK)
CREATE TEMPORARY TABLE `dim` (
    `id` BIGINT,
    `status` STRING,
    PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:postgresql://localhost:5432/postgres',
  'table-name' = 'dim',
  'username' = 'postgres',
  'password' = 'postgres'
);

-- Lookup join two tables twice with different filter, and union them together
SELECT
     s.id,
     s.name,
     s.txn_time,
     d.status
FROM `stream` AS `s` INNER JOIN `dim` FOR SYSTEM_TIME AS OF `s`.`txn_time` AS `d`
ON
     `s`.`id` = `d`.`id`
WHERE
     `d`.`status` = 'OK'
UNION ALL
SELECT
     s.id,
     s.name,
     s.txn_time,
     d.status
FROM `stream` AS `s` INNER JOIN `dim` FOR SYSTEM_TIME AS OF `s`.`txn_time` AS `d`
ON
     `s`.`id` = `d`.`id`
WHERE
     `d`.`status` = 'NOT_EXISTS';

We expect to get the following results:

1, Alice 2024-11-27 11:52:19.332, OK
2, Bob   2024-11-27 11:52:19.332, OK

However, the actual results we got were:

1, Alice, 2024-11-27 11:52:19.332, OK
2, Bob,   2024-11-27 11:52:19.332, OK
1, Alice, 2024-11-27 11:52:19.333, NOT_EXISTS
2, Bob,   2024-11-27 11:52:19.333, NOT_EXISTS

This is obviously wrong, since there are no statuses with value NOT_EXISTS in the dimension dim table.

And SQL plans reported on the ticket seem correct.

Abstract syntax tree:

LogicalUnion(all=[true])
:- LogicalProject(id=[$0], name=[$1], txn_time=[$2], status=[$4])
:  +- LogicalFilter(condition=[=($4, _UTF-16LE'OK')])
:     +- LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0, 2}])
:        :- LogicalProject(id=[$0], name=[$1], txn_time=[PROCTIME()])
:        :  +- LogicalTableScan(table=[[default_catalog, default_database, stream]])
:        +- LogicalFilter(condition=[=($cor0.id, $0)])
:           +- LogicalSnapshot(period=[$cor0.txn_time])
:              +- LogicalTableScan(table=[[default_catalog, default_database, dim]])
+- LogicalProject(id=[$0], name=[$1], txn_time=[$2], status=[$4])
   +- LogicalFilter(condition=[=($4, _UTF-16LE'NOT_EXISTS')])
      +- LogicalCorrelate(correlation=[$cor1], joinType=[inner], requiredColumns=[{0, 2}])
         :- LogicalProject(id=[$0], name=[$1], txn_time=[PROCTIME()])
         :  +- LogicalTableScan(table=[[default_catalog, default_database, stream]])
         +- LogicalFilter(condition=[=($cor1.id, $0)])
            +- LogicalSnapshot(period=[$cor1.txn_time])
               +- LogicalTableScan(table=[[default_catalog, default_database, dim]])

Optimized physical plan:

Calc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status])
+- Union(all=[true], union=[id, name, txn_time, status])
   :- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS status])
   :  +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id])
   :     +- Calc(select=[id, name, PROCTIME() AS txn_time])
   :        +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
   +- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS status])
      +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id])
         +- Calc(select=[id, name, PROCTIME() AS txn_time])
            +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])

And optimized execution plan:

Calc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status])
+- Union(all=[true], union=[id, name, txn_time, status])
   :- Calc(select=[id, name, txn_time, CAST('OK' AS VARCHAR(2147483647)) AS status])
   :  +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id])(reuse_id=[1])
   :     +- Calc(select=[id, name, PROCTIME() AS txn_time])
   :        +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
   +- Calc(select=[id, name, txn_time, CAST('NOT_EXISTS' AS VARCHAR(2147483647)) AS status])
      +- Reused(reference_id=[1])

I have also run the same query using our internal Flink engines or using MySQL as another database, but the results were the same. Thus fixing this bug in the open-source Apache Flink is important, all other enterprise Flink distributions will also benefit.

Understanding the Issue

From the quick look at the optimized execution plan, we can see that the first lookup join is reused. To make sure that this is not the source of the problem, we can disable the reuse by setting the following configuration:

SET table.optimizer.reuse-sub-plan-enabled=false;

We are still getting the wrong results.

To better debug the issue, it is a good idea to create and reproduce the issue in the tests. After looking into the planner tests I came up with a test case using the 'values' connector for both tables. For example, we can update the properties of the dimension table as below:

val dimTableId = TestValuesTableFactory.registerData(Seq(row(1L, "OK"), row(2L, "OK")))
val dimTableDDL =
  s"""
     | CREATE TABLE `dim` (
     |   `id` BIGINT,
     |   `status` STRING,
     |   PRIMARY KEY (`id`) NOT ENFORCED
     | ) WITH (
     |   'connector' = 'values',
     |   'data-id' = '$dimTableId'
     | )
     |""".stripMargin
tEnv.executeSql(dimTableDDL)

Hold and behold, the issue does not happen! Okay, this is good since sign, we can now try to find the reasons for differences when using different connectors.

Comparing Planner Transformations

We know that two queries behave differently when used with different connectors, to investigate further, let’s compare each query transformation. To do so, enable the debug logging or add log statements in the FlinkChainedProgram#optimize method.

(There are many optimization steps in the program they require a topic of their own)

Listing all the optimization steps and comparing each plan (with minor simplifications), we can see the following differences.

Logical Rewrite

The first difference is in the logical rewrite step.

Logical Rewrite rule for union all query using JDBC connector	Logical Rewrite rule for union all query using test 'values' connector
optimize 'logical_rewrite' cost 29 ms. original input: FlinkLogicalUnion(all=[true]) :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) : :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) : +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]) : +- FlinkLogicalSnapshot(period=[$cor0.txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status]) +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]) +- FlinkLogicalSnapshot(period=[$cor1.txn_time]) +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status]) optimize output: FlinkLogicalUnion(all=[true]) :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) : :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) : +- FlinkLogicalSnapshot(period=[$cor0.txn_time]) : +- FlinkLogicalCalc(select=[id]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]]], fields=[id, status]) +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- FlinkLogicalSnapshot(period=[$cor1.txn_time]) +- FlinkLogicalCalc(select=[id]) +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]]], fields=[id, status])	optimize 'logical_rewrite' cost 29 ms. original input: FlinkLogicalUnion(all=[true]) :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) : :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) : +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]) : +- FlinkLogicalSnapshot(period=[$cor0.txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status]) +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]) +- FlinkLogicalSnapshot(period=[$cor1.txn_time]) +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status]) optimized output: FlinkLogicalUnion(all=[true]) :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) : :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) : +- FlinkLogicalSnapshot(period=[$cor0.txn_time]) : +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status]) +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- FlinkLogicalSnapshot(period=[$cor1.txn_time]) +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]) +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status])

Logical Rewrite rule for union all query using JDBC connector

Logical Rewrite rule for union all query using test 'values' connector

optimize 'logical_rewrite' cost 29 ms.

original input:

FlinkLogicalUnion(all=[true])
:- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
:  +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
:     :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
:     :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
:     +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))])
:        +- FlinkLogicalSnapshot(period=[$cor0.txn_time])
:           +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status])
+- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
   +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
      :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
      :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
      +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))])
         +- FlinkLogicalSnapshot(period=[$cor1.txn_time])
            +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status])

optimize output:

FlinkLogicalUnion(all=[true])
:- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
:  +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
:     :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
:     :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
:     +- FlinkLogicalSnapshot(period=[$cor0.txn_time])
:        +- FlinkLogicalCalc(select=[id])
:           +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]]], fields=[id, status])
+- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
   +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
      :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
      :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
      +- FlinkLogicalSnapshot(period=[$cor1.txn_time])
         +- FlinkLogicalCalc(select=[id])
            +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]]], fields=[id, status])

optimize 'logical_rewrite' cost 29 ms.

original input:

FlinkLogicalUnion(all=[true])
:- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
:  +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
:     :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
:     :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
:     +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))])
:        +- FlinkLogicalSnapshot(period=[$cor0.txn_time])
:           +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status])
+- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
   +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
      :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
      :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
      +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))])
         +- FlinkLogicalSnapshot(period=[$cor1.txn_time])
            +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim]], fields=[id, status])

optimized output:

FlinkLogicalUnion(all=[true])
:- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
:  +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
:     :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
:     :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
:     +- FlinkLogicalSnapshot(period=[$cor0.txn_time])
:        +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))])
:           +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status])
+- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
   +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
      :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
      :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
      +- FlinkLogicalSnapshot(period=[$cor1.txn_time])
         +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))])
            +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status])

Well, what is going on here?

We have the same logical input, but the optimized results are different. After discussing with my colleagues, I realized that the JDBC connector supports filter push down capability. Because of this, the Flink SQL planner pushes the filter condition down to the table level when using the JDBC connector. This is not the case for the test values connector, and the filter/where condition is kept on the calculated logical expression that will be applied after the table scan.

Note

The filter push down optimization is important since it allows us to filter the data at the source level, reducing the amount of data transferred over the network, improving performance of the SQL queries.

Okay, this is fine. Let’s check the next difference.

Physical Optimization

The second difference happens in the physical optimization step.

Physical Optimization rule for union all query using JDBC connector	Physical Optimization rule for union all query using test 'values' connector
optimize 'physical' cost 29 ms. original input: FlinkLogicalCalc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status]) +- FlinkLogicalUnion(all=[true]) :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) : :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) : +- FlinkLogicalSnapshot(period=[$cor0.txn_time]) : +- FlinkLogicalCalc(select=[id]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]]], fields=[id, status]) +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- FlinkLogicalSnapshot(period=[$cor1.txn_time]) +- FlinkLogicalCalc(select=[id]) +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]]], fields=[id, status]) optimized output: Calc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status]) +- Union(all=[true], union=[id, name, txn_time, status]) :- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id], upsertKey=[[0]]) : +- Calc(select=[id, name, PROCTIME() AS txn_time]) : +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id], upsertKey=[[0]]) +- Calc(select=[id, name, PROCTIME() AS txn_time]) +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])	optimize 'physical' cost 29 ms. original input: FlinkLogicalCalc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status]) +- FlinkLogicalUnion(all=[true]) :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) : :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) : +- FlinkLogicalSnapshot(period=[$cor0.txn_time]) : +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status]) +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner]) :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time]) : +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- FlinkLogicalSnapshot(period=[$cor1.txn_time]) +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]) +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status]) optimized output: Calc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status]) +- Union(all=[true], union=[id, name, txn_time, status]) :- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status]) : +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[status=_UTF-16LE'OK', id=id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))], select=[id, name, txn_time, id], upsertKey=[[0]]) : +- Calc(select=[id, name, PROCTIME() AS txn_time]) : +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name]) +- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status]) +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[status=_UTF-16LE'NOT_EXISTS', id=id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))], select=[id, name, txn_time, id], upsertKey=[[0]]) +- Calc(select=[id, name, PROCTIME() AS txn_time]) +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])

Physical Optimization rule for union all query using JDBC connector

Physical Optimization rule for union all query using test 'values' connector

optimize 'physical' cost 29 ms.

original input:

FlinkLogicalCalc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status])
+- FlinkLogicalUnion(all=[true])
   :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
   :  +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
   :     :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
   :     :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
   :     +- FlinkLogicalSnapshot(period=[$cor0.txn_time])
   :        +- FlinkLogicalCalc(select=[id])
   :           +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))]]], fields=[id, status])
   +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
      +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
         :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
         :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
         +- FlinkLogicalSnapshot(period=[$cor1.txn_time])
            +- FlinkLogicalCalc(select=[id])
               +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))]]], fields=[id, status])

optimized output:

Calc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status])
+- Union(all=[true], union=[id, name, txn_time, status])
   :- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
   :  +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id], upsertKey=[[0]])
   :     +- Calc(select=[id, name, PROCTIME() AS txn_time])
   :        +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
   +- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
      +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[id=id], select=[id, name, txn_time, id], upsertKey=[[0]])
         +- Calc(select=[id, name, PROCTIME() AS txn_time])
            +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])

optimize 'physical' cost 29 ms.

original input:

FlinkLogicalCalc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status])
+- FlinkLogicalUnion(all=[true])
   :- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
   :  +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
   :     :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
   :     :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
   :     +- FlinkLogicalSnapshot(period=[$cor0.txn_time])
   :        +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))])
   :           +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status])
   +- FlinkLogicalCalc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
      +- FlinkLogicalJoin(condition=[=($0, $3)], joinType=[inner])
         :- FlinkLogicalCalc(select=[id, name, PROCTIME() AS txn_time])
         :  +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
         +- FlinkLogicalSnapshot(period=[$cor1.txn_time])
            +- FlinkLogicalCalc(select=[id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))])
               +- FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, dim, filter=[]]], fields=[id, status])

optimized output:

Calc(select=[id, name, PROCTIME_MATERIALIZE(txn_time) AS txn_time, status])
+- Union(all=[true], union=[id, name, txn_time, status])
   :- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'OK':VARCHAR(2147483647)) AS status])
   :  +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[status=_UTF-16LE'OK', id=id], where=[=(status, _UTF-16LE'OK':VARCHAR(2147483647))], select=[id, name, txn_time, id], upsertKey=[[0]])
   :     +- Calc(select=[id, name, PROCTIME() AS txn_time])
   :        +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])
   +- Calc(select=[id, name, txn_time, CAST(_UTF-16LE'NOT_EXISTS':VARCHAR(2147483647)) AS status])
      +- LookupJoin(table=[default_catalog.default_database.dim], joinType=[InnerJoin], lookup=[status=_UTF-16LE'NOT_EXISTS', id=id], where=[=(status, _UTF-16LE'NOT_EXISTS':VARCHAR(2147483647))], select=[id, name, txn_time, id], upsertKey=[[0]])
         +- Calc(select=[id, name, PROCTIME() AS txn_time])
            +- TableSourceScan(table=[[default_catalog, default_database, stream]], fields=[id, name])

This transformation also looks fine. But let’s pay closer attention to understand what is going on here.

This step converts the join into the lookup join, adds additional information, e.g, joinType, select and lookup definitions. It adds id as a lookup key to both transformations. However, for the values connector plan (on the right) the lookup definition includes the status column as a lookup key. Plus, it also contains information about the filter condition in where definition.

But on the JDBC connector plan (on the left) does not include where definition and status in the lookup definition. This shouldn’t be a problem for JDBC or any other source that support filter pushdowns since the filter condition (here status column comparison) is pushed down in the table scan.

But still, it would be helpful to include the filter pushdowns in the lookup join definitions in query explanations.

Warning

This is also foreshadowing for the source of the bug.

At this point, I have conducted several debug sessions and discussions with my colleagues, but I still could not identify the root cause of the bug.

I was thinking maybe some other optimization rule is messing up the filter pushdowns, e.g., discards them. Thus, I tried to create minimal test case that only applies the relevant physical optimization rules. You can find the example file here.

While debugging and reading other test cases, I noticed that the values connector could simulate the filter pushdowns, by just adding filterable-fields property to the table creation.

By updating the dimension table create statement to:

val dimTableId = TestValuesTableFactory.registerData(Seq(row(1L, "OK"), row(2L, "OK")))
val dimTableDDL =
  s"""
     | CREATE TABLE `dim` (
     |   `id` BIGINT,
     |   `status` STRING,
     |   PRIMARY KEY (`id`) NOT ENFORCED
     | ) WITH (
     |   'connector' = 'values',
     |   'filterable-fields' = 'id;status',
     |   'data-id' = '$dimTableId'
     | )
     |""".stripMargin

tEnv.executeSql(dimTableDDL

We have no the reproducible test case, and no need to depend on the databases. Okay good, but we are still have no clue about the root cause of the bug.

Continuing debugging, I added breakpoint on the StreamPhysicalUnionRule#440 and noticed that the logical FlinkLogicalJoin relations for both inputs are the same.

this = StreamPhysicalUnionRule @13203
rel = FlinkLogicalUnion @13204
▾ union = FlinkLogicalUnion @13204
- id = 888
- ▾ inputs = RegularImmutableList @13218
  - ▾ 0 = RelSubset @13227
    - id = 876
    - ▾ best = FlinkLogicalCalc @13232
      - id = 875
      - ▾ input = RelSubset @13237
        
        id = 874
        
        ▾ best = FlinkLogicalJoin @13242
        
        id = 873
        
        ▸ joinType = JoinRelType @13255
        
        ▸ condition = RexCall @13256
        
        ▸ joinInfo = JoinInfo @13258
        
        ▸ left = RelSubset @13259
        
        ▸ right = RelSubset @13260
  - ▾ 1 = RelSubset @13228
    - id = 886
    - ▾ best = FlinkLogicalCalc @13248
      - id = 885
      - ▾ input = RelSubset @13237
        
        id = 874
        
        ▾ best = FlinkLogicalJoin @13242
        
        id = 873
        
        ▸ joinType = JoinRelType @13255
        
        ▸ condition = RexCall @13256
        
        ▸ joinInfo = JoinInfo @13258
        
        ▸ left = RelSubset @13259
        
        ▸ right = RelSubset @13260

Somehow the SQL optimizer finds the best plan (cheapest cost plan) for the union with both logical joins equivalent.

Indeed, if we print the logical union relation before converting to the physical union, System.out.println(union.explain()):

Union: FlinkLogicalUnion(all=[true])
  FlinkLogicalCalc(subset=[rel#876:RelSubset#19.LOGICAL.any.None: 0.[NONE].[NONE]], select=[id, name, CAST('OK' AS VARCHAR(2147483647)) AS status])
    FlinkLogicalJoin(subset=[rel#874:RelSubset#18.LOGICAL.any.None: 0.[NONE].[NONE]], condition=[=($0, $2)], joinType=[inner])
      FlinkLogicalTableSourceScan(subset=[rel#867:RelSubset#14.LOGICAL.any.None: 0.[NONE].[NONE]], table=[[default_catalog, default_database, stream]], fields=[id, name])
      FlinkLogicalSnapshot(subset=[rel#872:RelSubset#17.LOGICAL.any.None: 0.[NONE].[NONE]], period=[$cor0.txn_time])
        FlinkLogicalCalc(subset=[rel#870:RelSubset#16.LOGICAL.any.None: 0.[NONE].[NONE]], select=[id])
          FlinkLogicalTableSourceScan(subset=[rel#868:RelSubset#15.LOGICAL.any.None: 0.[NONE].[NONE]], table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'OK':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")]]], fields=[id, status])
  FlinkLogicalCalc(subset=[rel#886:RelSubset#24.LOGICAL.any.None: 0.[NONE].[NONE]], select=[id, name, CAST('NOT_EXISTS' AS VARCHAR(2147483647)) AS status])
    FlinkLogicalJoin(subset=[rel#874:RelSubset#18.LOGICAL.any.None: 0.[NONE].[NONE]], condition=[=($0, $2)], joinType=[inner])
      FlinkLogicalTableSourceScan(subset=[rel#867:RelSubset#14.LOGICAL.any.None: 0.[NONE].[NONE]], table=[[default_catalog, default_database, stream]], fields=[id, name])
      FlinkLogicalSnapshot(subset=[rel#872:RelSubset#17.LOGICAL.any.None: 0.[NONE].[NONE]], period=[$cor0.txn_time])
        FlinkLogicalCalc(subset=[rel#870:RelSubset#16.LOGICAL.any.None: 0.[NONE].[NONE]], select=[id])
          FlinkLogicalTableSourceScan(subset=[rel#868:RelSubset#15.LOGICAL.any.None: 0.[NONE].[NONE]], table=[[default_catalog, default_database, dim, filter=[=(status, _UTF-16LE'OK':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")]]], fields=[id, status])

We can see that the FlinkLogicalJoin for both inputs are exactly the same deep into the table scan. Both parts only filter the data if the status column is equal to the string OK.

This is good finding!

At this point, I spent extra time debugging the LogicalJoin optimization rules, but it didn’t lead to anything.

Tip

Now is also good time to update our tests and add unit test for then case when the filter condition are reversed. That is what happens if we filter for the non-existing status values on the first part of the union query?

But if both logical joins filter data matching the status 'OK', why do we get the following result with NOT EXISTS status rows?

1, Alice, 2024-11-27 11:52:19.332, OK
2, Bob,   2024-11-27 11:52:19.332, OK
1, Alice, 2024-11-27 11:52:19.333, NOT_EXISTS
2, Bob,   2024-11-27 11:52:19.333, NOT_EXISTS

Shouldn’t all four rows have status as OK?

Code Generation

Prospero Challenge: JIT Codegen with LLVM and AsmJit

2025-08-27T00:00:00+00:00

I wanted to explore and deep dive into the Just-In-Time (JIT) compilation and code generation topics. To get some hands-on experience, I was looking for a fun project and came across the Prospero Challenge from Matt Keeter, which was a perfect fit.

The source code for my implementation is available at morazow/prospero-rust.

The challenge is simple, you are given list of ~8k mathematical expressions that must be evaluated for each pixel (x, y) in an image (e.g., 1024x1024 size). The pixel is then colored black or white based on the sign of the final result.

The input is a series of operations that build up the expression:

# Text of a monologue from The Tempest
_0 const 2.95
_1 var-x
_2 const 8.13008
_3 mul _1 _2
_4 add _0 _3
_5 const 3.675
# ...

When rendered, these expressions produce an image as following:

You can read more about the challenge and see solutions from other developers on the Prospero site.

My Approach

Since this was a learning project for me rather than a pure optimization challenge, I took a multi-stage approach, starting with the simplest implementation and progressively adding different JIT backends.

A Baseline Virtual Machine (VM)

I started by implementing a Virtual Machine (VM). This approach is straightforward and served as a baseline for verifying the correctness of the challenge.

JIT with LLVM

For my first Just-In-Time (JIT) compilation backend, I chose the powerful and popular LLVM framework. It is a portable, cross-platform ecosystem with a mature optimization steps. LLVM also provides a rich set of intrinsic functions (e.g., llvm.sqrt.f64 for square roots), which simplifies code generation.

To connect Rust frontend with LLVM, I used the inkwell crate, it helped to generate LLVM Intermediate Representation (IR) from project’s Abstract Syntax Tree (AST).

Low-Level JIT with AsmJit

Next, I wanted to experiment with AsmJit. It’s a lightweight C++ library designed for low-latency machine code generation, supporting X86, X86_64, and AArch64 architectures.

This part of the project required integrating C++ code with Rust. I built a Foreign Function Interface (FFI) bridge that allowed the AsmJit compiler to consume the Rust AST and generate native code.

Conclusion

This was a fun exercise for learning the fundamentals of just-in-time compilation, and progressing from a high-level VM to low-level machine code generation.

A special thanks to Matt Keeter for creating such an inspiring challenge!

Future Work

While I’m happy with the result, there are always more things to explore:

AST-level optimizations: Implementing duplicate expression elimination before code generation.
AsmJit enhancements: Adding a constant pool for the AsmJit backend (e.g., using a .data section) to reduce code size and improve performance.
Parallelism: Parallelizing the image rendering by processing tiles in multiple threads.
A C++ version: Re-implementing the project in C++ to avoid FFI in AsmJit and to directly use LLVM C++ API.

Finding Semicolons: Examples From 1BRC Submissions

2024-02-11T00:00:00+00:00

Gunnar Morling launched One Billion Row Challenge (1BRC) in the beginning of the year. The goal is to calculate temperature aggregates (min, max, sum) of weather stations. The data is one billion rows of measurements in <string: station>;<double: temperature> format.

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6
Bridgetown;26.9
Istanbul;6.2
Roseau;34.4
Conakry;31.2
Istanbul;23.0

The station names are arbitrary length strings and temperatures are of X.XX or XX.X formatted double values.

Submitted solutions were golden, there were many interesting, optimized submissions. I enjoyed reading them and trying to understand how they work and why they are fast. If only they were easier to understand :-)

Parsing

One of the tasks is to parse the input data, separate station name and its temperature value. For that we have to find the location of semicolon ';' in the each input line!

private static final byte SEMICOLON = ';';

private int findSemicolonPosition(byte[] line) {
    for (int i = 0; i < line.length; i++) {
        if (line[i] == SEMICOLON) {
            return i;
        }
    }
    return -1;
}

Linear scan to find the semicolon position in byte array

Apparently, simple task of finding a byte character in a given string could be heavily optimized.

In this blog I am going to look into two options that were used in 1BRC submissions, the first is “SIMD Within A Register (SWAR)” technique, and the second is to use Java Vector API. Both of these techniques take advantage of processing multiple data on a single instruction.

SWAR

SWAR is technique to process multiple bytes at once taking advantage of 64-bit processor CPU architectures.

The idea is to load multiple bytes into single 64-bit register and then perform bitwise operations to find the index of the matching byte. On little-endian machines, we want the index of the first matching byte from the right end of the register, since little-endian machines reverse the bytes when a word is loaded into a register.

Mainly we are looking for the following function, indexOfFirstMatched(word, pattern):

                                      /
                                     /  0, word = XXXXXXXXXXXXXXOO
                                    |   1, word = XXXXXXXXXXXXOONN
                                    |   2, word = XXXXXXXXXXOONNNN
                                   /    3, word = XXXXXXXXOONNNNNN
indexOfFirstMatched(word, 0xOO) = <     4, word = XXXXXXOONNNNNNNN
                                   \    5, word = XXXXOONNNNNNNNNN
                                    |   6, word = XXOONNNNNNNNNNNN
                                    |   7, word = OONNNNNNNNNNNNNN
                                     \  8, word = NNNNNNNNNNNNNNNN // return byte length
                                      \                            // if no match is found

The OO denotes the match byte, NN denotes a nonzero byte, and XX denotes a byte that maybe zero or nonzero. If no match is found, the function returns the length of word.

This technique perfectly fits for finding locations of semicolons in 1BRC problem during parsing of each line. Most of the submissions used technique from Richard Startin’s “Finding Bytes” blog post.

For example, Thomas Würthinger early submission (slightly modified by me):

private static int findDelimiter(long word) {
    long input = word ^ 0x3B3B3B3B3B3B3B3BL;
    long match = (input & 0x7F7F7F7F7F7F7F7FL) + 0x7F7F7F7F7F7F7F7FL;
    match = ~(match | input | 0x7F7F7F7F7F7F7F7FL);
    return Long.numberOfTrailingZeros(match) >>> 3;
}

However, there is improved version of above approach that works in little-endian architectures. An example of this is from Roy van Rijn early submission (slightly modified by me):

private static int firstAnyPattern(long word) {
    final long match = word ^ 0x3B3B3B3B3B3B3B3BL;
    long mask = (match - 0x0101010101010101L) & ~match & 0x8080808080808080L;
    return Long.numberOfTrailingZeros(mask) >>> 3;
}

Both of these methods find the index of first byte in the 64-bit long word that matches the semicolon ';' pattern encoded as 0x3B3B3B3B3B3B3B3BL.

To learn more why this works, please read Richard Startin’s explanation and check the visuals from Wojciech Muła’s “SWAR find any byte from set” post.

Finding 0-Byte

These optimizations are used in many applications that need to find the index of '\0' — characters in strings. The strlen function in C, instead of checking each byte separately, could load multiple bytes as word to check for the matching byte pattern.

The early versions of the SWAR algorithm was presented by Leslie Lamport in his paper titled “Multiple byte processing with full-word instructions” in 1975.

It’s a neat hack, and it’s more useful now than it was then for two reasons. The obvious reason is that word size is larger now, with many computers having 64-bit words. The less obvious reason is that conditional operations are implemented with masking rather than branching. Instead of branching around the operation when the condition is not met, masks are constructed so the operation is performed only on those data items for which the condition is true. Branching is more costly on modern multi-issue computers than it was on the computers of the 70s.

The Hacker’s Delight book, in Chapter 6 “Find First 0-Byte”, describes both of the above approaches using 32-bits words. The first method is attributed to Leslie Lamport because he uses similar tricks in his paper. The second method was proposed by Alan Mycroft on comp.arch newsgroup in 1987.

Let’s try to understand the Mycroft’s version.

private static int firstFirstSemicolon(long word) {
    long match = word ^ 0x3B3B3B3B3B3B3B3BL;
    long mask = (match - 0x0101010101010101L) & ~match & 0x8080808080808080L;
    return Long.numberOfTrailingZeros(mask) >>> 3;
}

0x3B is our pattern, it is ';' in ASCII
word ^ 0x3B will set all matching ';' bytes in word to 0x00
(x - 0x01) converts all 0x00 bytes to 0xFF, sets the highest bit of a byte to 1
- It converts the set of { 0x00, 0x81, 0x82, 0x83, ..., 0xFF } bytes into { 0xFF, 0x80, 0x81, 0x82, ..., 0xFE } set.
~x converts all 0x00 bytes to 0xFF, sets the highest bit of a byte to 1
- Similarly, this instruction converts { 0x00, 0x01, 0-x02, 0x03, ..., 0xFF } byte set into { 0xFF, 0xFE, 0xFD, 0xFC, ..., 0x80 } set.
(x - 0x01) & ~x will retain the highest set bit of byte, only if the byte is 0x00
- Since both above input sets contain 0x00, applying bitwise AND on both sets will keep the highest bit set only for the 0x00 byte.
0x80 then zeros all bits except the highest bit of each byte
By counting number of trailing zeros, we find the index of first 0x00 byte

This is nice explanation from “Detects zero bytes inside a 32 bit integer” article.

Java Vector API

The second optimization is to use the Java Vector API for finding index of the first matching pattern in a byte array.

Java Vector API is a preview feature in Java 21 that enable developers to take advantage of Single Instruction Multiple Data (SIMD) processors. Instead of depending on Java autovectorization, by using Vector API you tell the compiler to take advantage of the SIMD instructions.

To see how to use Java Vector API, let us calculate the norm of two arrays. The scalar implementation will be as following:

// Assuming arrays are of the same length
void scalarNormComputation(float[] a, float[] b, float[] c) {
    for (int i = 0; i < a.length; i++) {
        c[i] = (a[i] * a[i] + b[i] * b[i]) * -1.0f;
    }
}

And a possible vectorized implementation:

static final VectorSpecies<Float> SPECIES = FloatVector.SPECIES_PREFERRED;

void vectorNormComputation(float[] a, float[] b, float[] c) {
    int i = 0;
    int upperBound = SPECIES.loopBound(a.length);
    for (; i < upperBound; i += SPECIES.length()) {
        var va = FloatVector.fromArray(SPECIES, a, i);
        var vb = FloatVector.fromArray(SPECIES, b, i);
        var vc = va.mul(va).add(vb.mul(vb)).neg();
        vc.intoArray(c, i);
    }
    for (; i < a.length; i++) {
        c[i] = (a[i] * a[i] + b[i] * b[i]) * -1.0f;
    }
}

At the end, we perform calculation for the leftover elements that are not included in the vectorized loop.

Similar to norm computation, we can check equality (single instruction) of semicolon ; on multiple bytes at once using Vector API.

An example from Yevhenii Melnyk submission (slightly modified by me):

private static final byte SEMICOLON = ';';
private static final VectorSpecies<Byte> BYTE_SPECIES = ByteVector.SPECIES_PREFERRED;
private static final int BYTE_SPECIES_BYTE_SIZE = BYTE_SPECIES.vectorByteSize();
private static final Vector<Byte> SEMICOLON_VECTOR = BYTE_SPECIES.broadcast(SEMICOLON);

// Finds the position of first semicolon in the given byte array
private static int findDelimiter(BufferedFile file, int startPos) {
    int position = startPos;
    int vectorLoopBound = startPos + BYTE_SPECIES.loopBound(file.bufferLimit - startPos);
    for (; position < vectorLoopBound; position += BYTE_SPECIES_BYTE_SIZE) {
        var vector = ByteVector.fromArray(BYTE_SPECIES, file.buffer, position);
        var comparisonResult = vector.compare(VectorOperators.EQ, SEMICONLON_VECTOR);
        if (comparisonResult.anyTrue()) {
            return position + comparisonResult.firstTrue();
        }
    }

    while (file.buffer[position] != SEMICOLON) {
        position++;
    }

    return position;
}

Conclusion

I have also run these methods through JMH benchmarks. GitHub repository with the code and benchmarks is morazow/java-simd-benchmarks.

The benchmark evaluates the average running time (lower is faster) of each method on randomly generated 100K and 100M measurements data.

Benchmark                                         (filename)  Mode  Cnt       Score       Error  Units
FindingSemicolonBenchmark.linearScan   measurements-100K.txt  avgt   10     831,308 ±     5,847  us/op
FindingSemicolonBenchmark.swarLamport  measurements-100K.txt  avgt   10     837,476 ±     6,740  us/op
FindingSemicolonBenchmark.swarMycroft  measurements-100K.txt  avgt   10     755,864 ±     6,198  us/op
FindingSemicolonBenchmark.vectorAPI    measurements-100K.txt  avgt   10      41,236 ±     0,827  us/op
FindingSemicolonBenchmark.linearScan   measurements-100M.txt  avgt   10  850736,385 ± 13573,874  us/op
FindingSemicolonBenchmark.swarLamport  measurements-100M.txt  avgt   10  847787,746 ± 24557,818  us/op
FindingSemicolonBenchmark.swarMycroft  measurements-100M.txt  avgt   10  767232,577 ± 19555,154  us/op
FindingSemicolonBenchmark.vectorAPI    measurements-100M.txt  avgt   10   42925,814 ±  1993,941  us/op

Benchmark is run using OpenJDK 21 Temurin, on Apple M2 Pro with 12 (8 performance and 4 efficiency) CPU cores and 16 GB memory.

The linear scan method having similar running time as first SWAR method could be explained by the fact that branch predictor learning the benchmark data. The Java Vector API implementation runs the fastest.

In this post we only looked at a couple of the optimizations. There are many other interesting techniques that were used by the participants. The challenge was fun and great learning experience.

Thanks a lot Gunnar for your efforts and time organizing it!

References

Finding Bytes in Arrays by Richard Startin
SWAR find any byte from set by Wojciech Muła
Speed up strlen using SWAR in x86-64 Assembly StackOverflow Question
Bithacks: Determine if a word has a zero byte by Sean Eron Anderson
Multiple Byte Processing with Full-Word Instructions by Leslie Lamport
Detects zero bytes inside a 32 bit integer by Stephan Brumme

Understanding Java Memory Model

2023-10-13T00:00:00+00:00

These are notes taken to better understand the Java Memory Model, which I now publish as a blog post.

Programming language memory models, such as the Java Memory Model, attempt to define the behavior of multi-threaded programs. These specifications help to reason about code execution in a concurrent environment, even when the code runs on different hardware architectures or undergoes numerous compiler optimizations.

For example, given the following multi-threaded program:

Thread I	Thread II
`x = 1; done = 1;`	`while (done == 0) { / / } System.out.println(x);`

Initially, all variables are set to zero, and each thread runs in its own processor. Can we reason about the output of the program?

It appears that the output depends on the hardware and compiler optimizations. On x86 architecture, the assembly version of this code will always print 1. However, on ARM architecture, it may print 0. Additionally, compiler optimizations can cause this program to either print zero or enter an infinite loop.

As a programmer, it would be frustrating if programs don’t work on new hardware (mobile devices, cloud servers) or with new compilers. Thus, high-level programming language memory models are defined to provide a set of guarantees that programmers, while writing code in that language, can rely upon.

But first, let’s understand what guarantees are provided by the hardware.

Hardware Guarantees

Let us imagine we are writing an assembly code for our multiprocessor computer. What kind of guarantees could we expect from the hardware?

Sequential Consistency

From Leslie Lamport’s 1979 paper:

The customary approach to designing and proving the correctness of multiprocess algorithms for such a computer assumes that the following condition is satisfied: the result of any execution is the same as if the operations of all the processors were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program. A multiprocessor satisfying this condition will be called sequentially consistent.

This definition is natural to a programmer. It states that operations will be executed in the order they appear in a written program, and threads will be interleaved in some order.

For example, given the following program with two threads:

Thread I	Thread II
`x = 1; y = 1;`	`r1 = y; r2 = x;`

We could expect the following six outcomes from the above program in a sequentially consistent hardware, with interleaving of each thread operations.

x = 1;
y = 1;
            r1 = y; // 1
            r2 = x; // 1

r1 = y; // 0
r2 = x; // 0
            x = 1;
            y = 1;

x = 1;
    r1 = y; // 0
y = 1;
    r2 = x; // 1

x = 1;
            r1 = y; // 0
            r2 = x; // 1
y = 1;

r1 = y; // 0
            x = 1;
            y = 1;
r2 = x; // 1

    r1 = y; // 0
x = 1;
    r2 = x; // 1
y = 1;

As you can see, in the sequential consistent model the execution result where r1 = 1, r2 = 0 is not possible.

We can envision this model where all processors are directly linked to a single shared memory. In this case, there are no caches involved each write or read operation goes straight to the memory.

Sequential Consistent Hardware Model. Image © Russ Cox - Hardware Memory Models.

As we’ll see, modern hardware designs often give up on strict sequential consistency for performance reasons.

x86 Total Store Order (TSO)

The modern x86 architecture memory model is based on the following hardware structure.

x85 Architecture. Image © Russ Cox - Hardware Memory Models.

In this model, each write is queued in first-in, first-out (FIFO) order before being written to the shared memory. Similarly, each read first checks the local queue and then queries the shared memory. The local queue is flushed to the shared memory in FIFO order, ensuring that each write is applied in the same execution order in the processor.

This results in total store order (TSO). Once write reaches the shared memory, each next read sees it until it’s overwritten or buffered in the local write queue.

ARM Relaxed Memory Order

ARM processors have weaker memory models.

ARM Architecture. Image © Russ Cox - Hardware Memory Models.

As depicted in the picture, each processor has its own copy of memory. Each write propagates independently to other processors and there is a possibility of write reordering. Additionally, a read can be delayed until it’s needed or until after a write.

In ARM hardware model, there is no total store order. It only provides total order for writes on a single memory location (coherence) that we will see later.

Data Race Free Sequential Consistency (DRF-SC)

In their 1990 paper called Weak Ordering — A New Definition, Sarita Adve and Mark Hill introduced a synchronization model known as the data-race-free (DRF) model.

This model assumes that there are distinct hardware memory synchronization operations, and memory read or write operations can be rearranged between these synchronization operations. However, reads and writes must not be moved across the synchronization operations.

A program is considered data-race-free if any two accesses to the same memory location from two different threads are either both read operations or are separated by a synchronization operation that forces one to happen before the other.

Well, this provides an agreement between hardware and software, given a data-race-free program, the hardware will execute it in a sequentially consistent manner. The above paper provides a proof, even for weaker ARM hardware, that the hardware will appear sequentially consistent to a data-race-free programs. This guarantee is abbreviated as Data Race Free Sequential Consistency (DRF-SC).

So far, we have been assuming that we are programming in an assembly language that is close to the hardware. Let’s now examine the guarantees offered by high-level programming language, Java’s memory model.

Java Memory Model

In the preceding section, we learned that if a programming language offers synchronization mechanisms to coordinate different threads, we can use these to create data-race-free (DRF) programs. Data-race-free multi-threaded program operations could be arbitrarily interleaved, and the outcome of this program can be explained by some sequential consistent execution, as if the operations are run on a single processor.

Java offers various options for synchronization mechanisms to develop DRF programs. It’s crucial to understand that these are “synchronizing instructions” which establish a happens-before relationship between code executed on one thread and code executed on another.

The main Java synchronization operations are:

The creation of a thread happens before the first action in the thread.
Unlock of mutex m happens before any following lock of m.
Write to volatile variable v happens before any following read of v.

Of course, there are more synchronization operations in Java, but in this blog
we will focus on `volatile` since it demonstrates the main idea.

Happens Before

Java Memory Model (JMM) specifies which outcomes are permitted by the Java language. The outcomes are results of executions containing different orderings of operations of entire program.

We could arrange all operations, including lock, unlock, volatile write, and volatile read, in some interleaving order. Then, as an example, we could write to a volatile variable v and subsequently later in the ordering read v, which observes that write. This creates a happens-before edge on that particular execution.

These edges define whether an execution has a data race; if there is no data race, then the execution behaves in a sequentially consistent manner.

This conforms to the definition of DRF-SC.

Two events occurring on separate processors and not ordered by the
happens-before relationship may happen at the same moment; the exact order is
unclear. We refer to them as executing concurrently. A data race occurs when a
write to a variable executes concurrently with a read or another write of the
same variable.

We should remark two important points:

The happens-before edges also synchronize the rest of programs, ordering ordinary operations across threads.
The happens-before edges are not established by locking or unlocking different mutexes, or accessing different volatile variables.

Compiler Optimizations

These JMM rules define which optmizations are allowed by the compilers.

Given the following program with two threads, which instruction reorderings are allowed?

Multi-threaded program with happens-before edges.

These two orderings are allowed.

Write on y could be moved before write on volatile v since it doesn’t break happens-before (HB) edge and read on y can observe write on y with race.

Similarly, read on x could be moved after read on volatile v since it could observe result of write on x with race.

These two orderings are not allowed.

Both of these orderings break the happens-before edge.

For the first case, implementations cannot know if there is any read on x that should observe write on x before moving. Similarly, in the second case, implementations are unable to determine if the moved read on y should observe any preceding write to y.

Litmus Tests

In this section, we are going to run several litmus tests to evaluate how different models behave. These tests provide an answer to the question if certain outcome is possible or not under specific model.

In these examples, we assume that each shared variable starts with zero and each thread runs in its own dedicated processor. The rN is a thread-local variable, and we check if a thread-local result is possible at the end of execution.

Message Passing

Can this program see the following result r1 = 1, r2 = 0?

Thread I	Thread II
`x = 1; y = 1;`	`r1 = y; r2 = x;`
- On sequentially consistent hardware: no - On x86-TSO: no - On ARM: yes - Java language (plain): yes - Java language (volatile y): no

This outcome isn’t possible in a sequentially consistent hardware model. We can imagine that each processor is directly connected to the shared memory, there are no caches, registers, or write buffers. Thus, if we see an update on x, then we should also see the update on y.

Similar reasoning applies to the x86-TSO model. The write buffer from a single processor is flushed in FIFO order, ensuring that the update on x should become visible if the update on y happens.

Please be aware that we are writing these programs in assembly like language, where each instruction is executed by the processor. For the ARM model, these instructions can be reordered. Writes can be run in a different order, resulting in the possibility of an outcome.

For the Java language, using plain variables, the outcome mentioned above is possible. The optimizing compilers can reorder updates on the x and y variables.

However, if we declare y as volatile, then write and read on a volatile variable produces a happens-before edge. This edge precludes the optimizing Java compiler from moving the update on x after write on y operation.

By making variable y volatile, the above outcome is not possible. Similar to x86-TSO, update on a volatile propagates all updates, including ordinary variables, before it into the memory. If read on y results in 1, then read on x must also result in 1.

Message Passing Test — Happens Before Edges

We could easily visualize the happens-before (HB) edges in the possible executions of the message passing test.


HB consistent, reads observe latest writes on the happens-before edge.	HB consistent, reads observe the initial values.

HB consistent, it is racy read. There is not happens-before edge between write on x and read on x. It is read via race. HB allows observing "unsynchronized" writes via race.	HB incosistent. We cannot use this particular execution to reason about program outcomes. This outcome is impossible.

As expected the outcome r1 = 1, r2 = 0 is not possible in Java when y is volatile variable.

Store Buffering

Can this program see the following result r1 = 0, r2 = 0?

Thread I	Thread II
`x = 1; r1 = y;`	`y = 1; r2 = x;`
- On sequentially consistent hardware: no - On x86-TSO: yes - On ARM: yes - Java language (plain): yes - Java language (volatile x and y): no

On sequentially consistent hardware, this outcome isn’t possible because instructions are executed in total store order. However, on x86-TSO, the outcome is possible because the write buffer from a processor may not have been flushed to shared memory.

Due to the reordering of instructions, it’s also possible to achieve the outcome in both ARM and Java programming languages (using plain variables).

By making both variables x and y volatile, the above outcome is not possible. There exists a happens-before edge between the write and read operations on both volatile variables, so one of the writes should be the first to be executed. Remember that reordering of instructions is not allowed across synchronization operations.

Load Buffering

Can this program see the following result r1 = 1, r2 = 1?

Thread I	Thread II
`r1 = x; y = 1;`	`r2 = y; x = 1;`
- On sequentially consistent hardware: no - On x86-TSO: no - On ARM: yes - Java language (plain): yes - Java language (volatile x or y): no

This outcome isn’t possible in sequentially consistent and x86-TSO modes because the instructions cannot be reordered.

On an ARM model and with Java using plain variables, the outcome is possible because reads can be delayed until after writes.

In Java, declaring one of the variables as volatile, the above outcome is not possible. Any write operation on a volatile variable creates a happens-before edge, which prevents the compiler from reordering the instructions.

Coherence

Can this program see the following result r1 = 1, r2 = 2, r3 = 2, r4 = 1?

Thread I	Thread II
`x = 1;`	`x = 2;`
Thread III	Thread IV
`r1 = x; r2 = x;`	`r3 = x; r4 = x;`
- On sequentially consistent hardware: no - On x86-TSO: no - On ARM: no - Java language (plain): yes - Java language (volatile x): no

Here is something that is not possible in the ARM model. This test checks if updates to a single memory location are observed in a different order.

The threads agree on the total order of writes to a single memory location. One of the writes overwrites the other, and all the hardware models agree on the order.

However, due to optimizing compilers, the read instructions on thread IV could potentially be reordered. While this is possible in Java with regular variables, it is not feasible with volatile variables.

Independent Reads of Independent Writes (IRIW)

Can this program see the following result r1 = 1, r2 = 0, r3 = 1, r4 = 0?

Thread I	Thread II
`x = 1;`	`y = 1;`
Thread III	Thread IV
`r1 = x; r2 = y;`	`r3 = y; r4 = x;`
- On sequentially consistent hardware: no - On x86-TSO: no - On ARM: yes - Java language (plain): yes - Java language (volatile x and y): no

This is similar to the coherence test, but with two distinct memory locations. Primarily, we’re verifying whether threads III and IV observe the updates on x and y in different orders.

On ARM, there is no total store order guarantee on different writes. Similarly, the optimizing compiler could reorder the r3 and r4 reads, making thread interleaving to produce the above outcome.

Adding volatile to both variables, the outcome isn’t possible since it creates a happens-before edge that prevents the compiler from reordering the reads.

JCStress Tests

All of the litmus tests are validated using the JCStress framework. You can find the GitHub repository here at github.com/morazow/jmm-litmus-tests.

Locks

Java locks also provide the ordering, as lock enter happens before lock exit, which is similar to the behavior of volatile write and read. However, they ensure mutual exclusion, preventing two threads from concurrently accessing the locked or synchronized section.

Conclusion

With blog I tried to summarize my understanding of the Java Memory Model.

We began by learning about the guarantees offered by the hardware models.
We learned that by using proper synchronization mechanisms to ensure data-race-free implementations, the program outcomes could be explained as though they are executed in sequentially consistent manner.
We looked into the Java Memory Model and learned how happens-before edges are established.
We also ran several litmus tests to better understand how different models behave.

Acknowledgements

I could not have understood this topic without the help of the following resources.

E in REPL

2018-07-29T00:00:00+00:00

06 August 2018 - Lagos, Portugal

Eval in REPL

There are many strategies when evaluating a source code in interpreted languages.

The most common and easy to implement method is a tree-walking interpreter. Interpreters working in this way just evaluate a provided Abstract Syntax Tree (AST). Usually there might be preceding steps to make optimizations such as rewriting or transforming an AST so that it is more suitable for repeated or recursive evaluation.

Other interpreters first convert the AST to bytecode. Bytecode is composed of opcodes, which are similar to mnemonics of assembly language. However, the bytecode needs to be emulated by a virtual machine that is part of interpreter. This approach can be more performant than a tree-walking interpreter evaluation.

However, some interpreters do not build an AST at all. The parser directly emits bytecode, and then it gets emulated by a virtual machine.

Yet, some programming languages parse a source code, build an AST and convert the AST to bytecode. But instead of emulating opcodes specified by bytecode, the VM compiles them into native machine code before executed them — just in time. Usually these are called JIT (just in time) interpreters or compilers.

Some interpreters recursively traverse the AST but convert specific branches of it into native code, then execute that branch just in time. Slight variation of this is where a particular branch is compiled to native code only after traversing it multiple times.

Evaluation in Real World Programming Language

Ruby started as a tree-walk interpreter, executing the AST while traversing it, until version 1.9. With version 1.9, they introduced a virtual machine. After that Ruby interpreter parses source code, builds an AST and then compiles the AST into bytecode, which gets interpreted by virtual machine.

Lua started out as an interpreter that compiles to bytecode without building an AST, and then the bytecode is executed in register-based virtual machine. However, with introduction of LuaJIT, the bytecode is compiled to highly-optimized machine code for several architectures.

Conclusion

In summary, it is a trade off between performance or portability. If you want a performant language it is better to choose a bytecode VM that JIT compiles to native code for different machine architectures. But, tree-walking interpreters are less performant but portable since you do not have to target different architecture, only evaluate an AST.

Resources

Many of this material was adopted from Thorsten Ball’s Writing An Interpreter book. Additionally, Crafting Interpreters by Robert Nystrom was great help. I thank them both dearly!

Using MultiGroupBy with Scalding

2014-11-14T00:00:00+00:00

18 November 2014 - Nuremberg

TL;DR

Scalding .groupBy and .join operations can be combined into single operation using MultiGroupBy from Cascading extension, which improves the job performance. Scalding job example using MultiGroupBy can be found here.

Introduction

Let’s imagine we have two data sources. The first data contains the purchase record of the users per time and per geographical State. This data is formatted as following, <user_id, timestamp, state, purchases>. The second data contains the user demographic information. For this particular example, it only contains user age, <user_id, age>

The main goal of this map reduce job is to count the number of purchases per state and per age group.

In Scalding, we can implement this job as,

class MultiGroupByExample1(args: Args) extends Job(args) {
  // ...

  val Purchases =
    Tsv(purchasesPath, ('USERID, 'TIMESTAMP, 'STATE, 'PURCHASE))
    .read

  val UserAges =
    Tsv(userAgesPath, ('USERID, 'AGE))
    .read

  val MyJob = Purchases
    .groupBy('USERID, 'STATE) { _.size('COUNT) }
    .joinWithSmaller('USERID -> 'USERID, UserAges)
    .groupBy('STATE, 'AGE) { _.sum[Int]('COUNT) }
    .write(Tsv(outputPath))
}

This is elegant and concise solution however it is not very efficient.

The Problem

In Scalding each .groupBy and .join operation introduces another map reduce phase. That is with the code above, data will be shuffled, sorted and reduced three times before finishing the computation. Therefore, when there are very big data to be processed, the overall job performance will be very inefficient.

Luckily we can do better!

MultiGroupBy Operation

The desired solution is to perform aggregation operations while joining two data sources. Fortunately, it can be achieved using MultiGroupBy operation. In the rest of this blog I will show how to use MultiGroupBy in Scalding by reducing the three steps from above job into single map reduce phase.

Recently I was reading tips for optimizing Cascading flows (at https://nathanmarz.com/blog/tips-for-optimizing-cascading-flows.html) and recalled Cascading extensions project, which I saw several months ago. It offers additional operations on top of Cascading. Here I will only show MultiGroupBy (maybe BloomJoin in some other blog post). It is great!

The API of MultiGroupBy is defined MultiGroupBy.java#L35-L55. It accepts two pipes, two fields definitions as joining fields, renamed join field(s) and aggregation operation. We will have to write Cascading multi buffer operation in Java, but it is worth the effort.

The updated Scalding job will be as below,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import com.liveramp.cascading_ext.assembly.MultiGroupBy

class MultiGroupByExample2(args: Args) extends Job(args) {
  // ...

  val MyJob =
    new MultiGroupBy(
      Array(UserAges, Purchases),
      Array(new Fields("USERID"), new Fields("USERID")),
      new Fields("USERID"),
      new MyMultiBufferOp(new Fields("STATE", "AGE", "COUNT"))
    )
    .discard('USERID)
    .write(Tsv(outputPath))
}

Because MultiGroupBy performs join operation, it keeps the join fields. Therefore, on line 13 we just discard ‘USERID column.

Please notice the smooth Scala/Scalding and Java/Cascading interop. new Fields(“USERID”) and ‘USERID are the same.

Next we write our multi buffer operation, MyMultiBufferOp.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
import com.liveramp.cascading_ext.multi_group_by.MultiBuffer;
import org.apache.commons.collections.keyvalue.MultiKey;

public class MyMultiBufferOp extends MultiBuffer {
    // ...
    @Override
    public void operate() {
        // First pipe: UserAges <USERID, AGE>
        Iterator<Tuple> userAges = getArgumentsIterator(0);
        if (!userAges.hasNext()) {
            return ;
        }
        Tuple userAgesTuple = userAges.next();
        int user_age = userAgesTuple.getInteger(1); // second field is age
        // Data structure to store the count
        MultiKey key = null;
        Map<MultiKey, Integer> countMap = new HashMap<MultiKey, Integer>();
        // Second pipe: Purchases <USERID, TIMESTAMP, STATE, PURCHASES>
        Iterator<Tuple> purchases = getArgumentsIterator(1);
        while (purchases.hasNext()) {
            Tuple purchasesTuple = purchases.next();
            int state = purchasesTuple.getInteger(2); // third column is state
            key = new MultiKey(state, user_age);
            if (countMap.containsKey(key)) {
                countMap.put(key, countMap.get(key) + 1);
            } else {
                countMap.put(key, 1);
            }
        }
        // We just calculated <STATE, AGE, COUNT> results stored in 'countMap'
        // Now we just have to emit COUNT, because we gave <STATE, AGE>
        // as grouping names when calling this buffer operation
        for (Map.Entry<MultiKey, Integer> entry : countMap.entrySet()) {
            key = entry.getKey();
            int state = (Integer) key.getKey(0);
            int age = (Integer) key.getKey(1);
            int count = entry.getValue();
            emit(new Tuple(state, age, count));
        }
    }
}

First, we obtain tuple iterators for the two data sources. Then we keep updating the hashmap HashMap(<state, age>, count) until exhausting iterators values. Finally, we emit the hashmap contents as results for this buffer operation.

You can find the full code here and here multi buffer operation. In order to test the MultiGroupBy example you will have to assembly fat jar and run it on Hadoop environment.

Conclusion

In find this kind of patterns, join after or before groupBy, a lot in our map reduce job chains. Using MultiGroupBy we achieved considerable performance increase. Additionally, it resulted in efficient cluster utilization.

I strongly believe this operation should be default in both Cascading and Scalding.

If you liked this post, you can click to Tweet it or follow me on Twitter!

Scalding TemplatedTsv And Hadoop Many Files Problem

2014-10-03T00:00:00+00:00

03 October 2014 - Nuremberg

TL;DR

If Scalding TemplatedTsv tap creates lots of output files, do a groupBy on template column(s) just before writing the tap.

1
2
3
4
pipe
  // some other etl
  .groupBy('STATE) { g => g.pass }
  .write(TemplatedTsv(baseOutputPath, "%02d", 'STATE))

You can find full pseudocode at the bottom of this page.

Background

My daily job involves writing Hadoop map reduce jobs. I use Scalding and Cascading. They are really really really awesome. I cannot recommend them enough.

Usually we have several chain of map reduce jobs running. One of the jobs performs daily aggregation of the incoming data. The result of this job is then used as input for other jobs that run weekly or monthly.

The Problem

Let’s imagine the input data is formatted as, <user_id, timestamp, state, transactions> That is we have data about user making transactions in particular timestamp (epoch) per place, which is geographical State.

The main goal of this job is to count the number of transactions user made each day per state. In Scalding that would be map and groupBy operations.

pipe
  .map('TIMESTAMP -> 'DAY) { ts: Long => ts % (3600 * 24) }
  .groupBy('USERID, 'DAY, 'STATE) { g => g.size('COUNT) }
  .write(Tsv(baseOutputPath))

One other requirement for the job is that it needs to store its results into /year/month/day/state/ partitions.

Depending on the incoming input data we need to partition the aggregated data. That is, all the transactions for a particular state should be in single bucket partition. Input data may not contain all states, we should not create folders for not existing states.

The Solution

To achieve the goal we can use TemplatedTsv tap from Scalding. Just change the Tsv tap with it,

  .write(TemplatedTsv(baseOutputPath, "%02d", 'STATE))

When running the job jar just give the base output path as --output /year/month/day/, and it will create state folders inside above path.

However, this approach will create lots files. Because the data is not organized in any way, each reducer will have data containing several states, reducers will create several files in the state folder.

This is very very bad for the next jobs in the chain if they use as input the results of above job. For instance, weekly running job will be very slow because of lots files it has to read.

Can we mitigate this problem somehow?

Yes, sure. When reducers are done processing the data and about to write, we want the data that reducer processed to be from one (or two) state at most. So it will create one or two files at most.

To achieve this, sort the data by state using Hadoop power before writing. In other words, we just add another groupBy operation in Scalding and do not perform any aggregation operation in that grouping.

.groupBy('STATE) { g => g.pass }
.write(TemplatedTsv(baseOutputPath, "%02d", 'STATE))

This solves many files problem by introducing another map reduce phase overhead.

However, there is another problem with this solution. Because the data is not balanced with respect to state, some reducers will process only records (which might be a lot) belonging to a single state and delay the whole process.

Now the problem at hand is that some reducers process considerably large percentage of the data while some others process very small percentage. Therefore, our next goal is to process the states with lots of data in parallel with several reducers instead of single reducer handling that state.

After analyzing the incoming data or the results of the previous aggregation jobs, we can determine the states containing large portion of data. And distribute their load to number of reducers (of our choice) with the following trick,

.map(('USERID, 'STATE) -> 'SORTER) { tuple: (String, Int) =>
  val modulo = tuple._2 match {
    case 6  => 5
    case 48 => 7
    case 37 => 3
    case _  => 1
  }
  tuple._1.hashCode % modulo
}
.groupBy('STATE, 'SORTER) { g => g.pass }
.discard('SORTER)
.write(TemplatedTsv(baseOutputPath, "%02d", 'STATE))

For instance, we redistribute the California (6) state’s data into five reducers. Therefore, instead of single reducer, five of them will be writing into output partition thus creating file smaller files.

Conclusion

TemplatedTsv is great. However, it creates lots of small output files, which affects negatively the performance of the next job on the chain. Fortunately, the number of files can be reduced by sorting the data according to template before writing the tap. Furthermore, if the data is skewed you can apply some tricks to balance the templated data among reducers. This adds overhead of another map reduce phase.

val pipeSource = Tsv(InputSource, ('USERID, 'TIMESTAMP, 'STATE, 'TRANSACTIONS))

val pipeETL = pipeSource
  .read
  .map('TIMESTAMP -> 'DAY) { ts: Long => ts % (3600 * 24) }
  .groupBy('USERID, 'DAY, 'STATE) { g => g.size[Int]('COUNT) }
  .map(('HASHCODE, 'STATE) -> 'SORTER) { tuple: (String, Int) =>
    val modulo = tuple._2 match {
      case 6  => 5
      case 48 => 7
      case 37 => 3
      case _  => 1
    }
    tuple._1.hashCode % modulo
  }
  .groupBy('STATE, 'SORTER) { g => g.pass }
  .discard('SORTER)
  .write(TemplatedTsv(baseOutputPath, "%02d", 'STATE))

If you liked this post, you can click to Tweet it or follow me on Twitter!

Madeira Adasyna Gezelenç

2012-05-17T00:00:00+00:00

2012 ýylyň 17-nji maýy - Lissabon

Ýene bir gezelenç, Portugaliýa degişli bolan, okeýanyň ortasynda ýerleşen ada Madeira. Bu gezek kursdaşym, dostym Vaidas Brundza bilen gitdim. Üç gün ýaly gowy edip adany aýlandyk, ýaman gowy tebigaty bar ekeni.

Ilkinji gün, uçar bilen gelen şäherimiz Funchal-a gezişdirdik, birnäçe ýerli tagamlary datdyk. Ikinji gün bolsa, awtoulag arenda alyp ada aýlanmaga gitdik. Şu ýerde okyjylara kömek bolar umudy bilen birnäçe zady bellemekçi, ilki bilen aeroportda awtoulag arenda almaň sebäbi iki esse baha pul tölemeli, iň gowysy awtobus bilen şäher merkezine gidip şol ýerden almak. Biz Europcar awtoulag arenda hyzmatyny ulanypdyk, arzan we amatly. Hawa näme indi elimizde ulag bar adany bir başyndan aýlandyk. Ilki bilen iň beýik depesine tarap ýola çykdyk.

Şeýlelik bilen adanyň ýarysyna diýen ýaly aýlandyk. Gyzykly gezelençleriň biri boldy.

Ýewropanyň iň günbatar çüňki, Cabo da Roca

2012-02-20T00:00:00+00:00

2012 ýylyň 20-nji fewraly - Lissabon

Şu gün irden oglanlar bilen Ýewropanyň iň günbatar çüňki bolan Cabo da Roca gitmegi maksat edinipdik. Hawa Lissabondan ol ýerik ýöräp ha gidip bolmaýar, iň amatlysy welisipet (tigir) bilen gitmek. Ilki Lissabondan Cascais-a çenli otly bilen gitdik. Ol ýerde hem mugt tigir berýän ýerler bar, şolaryň birinden tigirlerimizi alyşdyryp ýola düşdük.

Bäş oglan bolyp gidipdik, içinde iň erbet sürýäni men. (:D) Sebäbi tigir ulanmanyma eýýäm ýedi sekiz ýyl bolyp barýardy. Hatta ýolda iki gezek ýykyldym, birindi ýolda işleýän adamlary süsdüm.. Ýenede gezelenç gaty gyzykly geçdi. Portugaliýanyň (hatta ýewropanyň) günbatar kenaryndan, okaýanyň gyrasyndan tigir sürýädik. Ýolda käwagtlar depeleri çykmaly bolýar, ýadatýar. Ýöne gaýdyşyn has hezil, welikleri sürmek hökman däl, diňe ýönüňi gönülemek ýeterlikdi.

Cabo da Roca-a ýetip gowy dynç aldyk, okeýanyň gyrasynda otyryp iýip içdik. Daş töweregi birsalym aýlanyp yzymyza dolandyk. Ýolda bir plaja sowylyp ol ýerde hem bir salym okeýan tolkynlary synlap çägede oýnap bildik.

Portugaliýada gezip görer ýaly gaty kän ýer bar..

EMDC Winter Event

2012-02-15T00:00:00+00:00

15 February 2012 - Sintra

After traveling around the Europe, I have eventually arrived to Lisbon. There was an EMDC event at Sintra that I should attend. I was excited since it would be gathering whole actors of this program. Some guys from previous batch, guys from Barcelona and Lisbon tracks and of course professors, and some invited speakers were there.

Sintra was great place to have such an event. Meeting fellow EMDC-ers and talking, getting tips from seniors was another activity I enjoyed very much. Moreover, we had some scientific talks about Distributed Computing, how to write master’s thesis, etc. Another presentation was given about Stockholm, Sweden and KTH where we will be studying one semester.

One thing I really liked was, seniors wore t-shirts while they were giving presentation, and the tagline was

“no tagline yet, still running paxos”

maybe I will use it in this blog as a tagline.

Eurail Gezelenjiň Netijesi

2012-02-11T00:00:00+00:00

2012-njy ýylyň 11-nji fewraly - Lissabon

Şeýlelik bilen 15 günlik gezelenji hem gutardym. Käwagtlar ýadadym, üşedim ýöne şonda hem gaty gyzyklydy.

Birnäçe agza sorapdy wiza gerekmi ýa diňe Eurail bilet alsaň bolýamy diýip, “Wiza gerek”! Öňüräk mail bilen özüm hem soradym şu ýerik ýazjak, “The Eurail Pass is no visa; travelers are responsible for ensuring that they have all necessary travel documents and visas.” Ýagny bilet satyn alanyňyzdan soň hökman haýsam bolsa bir şengen ýurtlaryndan turist wiza alaýmaly. Ýöne ýene gowy habar, eger haýsam bolsa bir Türkmenistan raýaty Türkiýede (ýada başga haýsam bolsa bir ýewropa ýurdynda) alty aýdan köp ýaşan bolsa arkaýyn Interrail (bu Eurail-dan arzan) bileti satyn alyp, ýanyna pasaportyny we ikametini alyp arkaýyn gezip biler. Sebäbi Türkiýani hem ýewropa zonasynda diýip hasaplapdyrlar.

Kiçijek Maslahatlarym

Men gyşda gezdim, ýanyňyza ýyly eşikleri hökman alyň, ellik, şarf, şapka we ýene hökman alynmaly zatlaryň biri hem şypbyk, gündiz gezip ýadap agyr göwüşi çykaryp arkaýyn dynç almak üçin. Ikinji men ýanyma içine girilip ýatylýan ýorgan (adyny bilemok) alypdym, käwagtlar garaşmaly bolanda hezillik berdi. Bulardan başga hem karta, telefonyň maps application-y ulanýadym ýöne käwagt internet tapdyraýanok. Mysal üçin men 3G ulanmak üçin Italiýada bolan bir operator sim kart hem alypdym. Birem şäher hakynda azajyk hem bolsa öňünden maglumat edinmek peýdaly, wikipediýadan okamak ýeterlik. Sebäbi, mysal, Prague-da eger gyzyk çyrada geçseň tutaýsalar 60 Kron (~ 3euro) jerime bar. Iň möhümi hem gymmat baha zatlaryňyza göz gulak bolmak, pul, laptop, fotoaparat wş.

Hawa näçe çykdaýjy boldy? Men hasapladym, 15 günlik Eurail bilet hem içinde, 1300 euro töweregi. Esasy diňe suwinirlere köp sowdym. Iýip-içki üçin kän pul berenim ýadyma düşenok, diňe Italiýada bir ýerde betinden pizza başga bir ýerde bolsa pasta iýdim. Başga wagtlar diňe sandwiç we çaý alyp iýmegim ýeterlik bolýady. 4 sany hostelde galdym olaram gymmat däldi, gijelik 15 euro. Öňünden zanitleseňä hasam arzan bolýar.

Näçe sowan bolsam hem hiçhili gynanamok sebäbi gezelençden gaty lezzet aldym we meň üçin üýtgeşik tejribe boldy. Birnäçe dost edindim, köp nätanyş adamlar bilen sohbetdeş boldym we iki sany dünýäniň başga başga ýerlerinden bolanlaryň gürleşip maglumat paýlaşmagy gaty üýtgeşik..

Iň soňky netijäm, eger ýenede ekonomiki we wagt mümkinçiligim bolsa ýene gezmekçi..

Berlin we Amsterdam Gezelenç

2012-02-07T00:00:00+00:00

2012-njy ýylyň 07-nji fewraly - Amsterdam

Berlin

Gije sagat on töwerekleri Berlin wokzalyna geldim, göni öňünden reserve eden Meinengir hostelime gitdim, wokzalyň edil ýanynda ýerleşýärdi. 10euro bilen bir gijelik galdym. (Aslynda öz bahasy mundan azajyk gymmadyrak ýöne Eurail bileti bolanlara skidka edýäler)

Ýatyp dynjymy alyp ertesi gün Berlinde bolan birnäçe türkmen oglanlara jaň edip duşuşdym. Gunduz oglanlar bilen Alexanderplatz we töwereklerine aýlandym. Soň hem elbette diňe Berline mahsus bolan Berlin Diwarynyň (Berlin Wall) galyndylaryny görmäne gitdim. Diwaryň yzy häzirem dur, köp zatlary aýyrmandyrlar.

Birem Berliniň Metro sistemasy iň çylşyrymly bolany öýdýän, düşünýänçäm kelle çişdi.

Amsterdam

Iki gün Berlinde türkmen oglanlary bilen galyp gije Amsterdama tarap ýola düşdim. Irden sagat 11 töwerekleri gelip bir Inner Amsterdam hostela ýerleşdim.

Amsterdam ýeke söz bilen çözýär. Çynym özüňiz gelip gözüňiz bilen görmeli. Şäheri bet edip oňarypdyrlar, birnäçe kanal geçýär içinden. (Kartasyna seredip görüň) Birem ýolda gidip barýadym welin kanallaryň biri doňypdyr we adamlar içinde skating edip ýördüler.. (:D) Şäheriň ýene bir üýtgeşikligi her ýerde welikleriň bolmagy. Menä adamdan kän tigir barmyka öýtdüm. Ýörite ýollary zatlaram bar.

Edil galan hostelimiň ýanynda Van Gogh muzeýi bardy ýöne gitmedim, giriş üçin 14euro isleýärdiler, “Şuňa girenimden bir t-shirt alanym gowylaý” diýip iki sany t-shirtjik suwinir aldym.

Amsterdama gelip edilmeli köp zatlardan birnäçesi Coffeeshop-a gitmek, we meşhur Red Light Street-i gezmek.

Berlinde bir gün artykmaç galanym üçin bileti mundan soň ulanyp bilemokdym, eýýäm 15 gün doldy. Netijede Amsterdamdan Lissabona uçaga bilet aldym. Ýöne ýenede gezelenç gutaranokdy, öňümde iki günlik UNESCO-nyň goragy astyna alan Lissabon Sintra gezelençi bardy..

Zurich-Vienna-Prague Gezelenç

2012-02-05T00:00:00+00:00

2012 ýylyň 2-nji fewraly - Berlin

Zürich

Irden sagat 7 töwerekleri Milan wokzalyndan Zürich gidýän otla mündim. Öňümde üç sagatlyk ýol bardy. Milan-Zürich arasy gezelenç hasam owadandy sebäbi Şwisariýada bolan Alp daglarynyň içinden geçip gidýärdik. Şol wagt şu ýylyň ilkinji garyny hem gören wagtym boldy, howa hem azajyk sowap başlapdy öňki şäherlerdäki gezelençlere görä hasam üşäp başlapdym.

Zürich wokzalyna gelemde ilkinji gören zadym bagaj goýulýan ýer (kamera hraneniýa). Öňki şäherlerde näme üçindir görmedim ýa-da kellä gelmedi şu, derrew uly goşymy goýyp diňe fotoaparat bilen şäheri gezmäne gitdim. Ýörän ýolymyň gyrasynda kartajyk bardy, seretsem ýakynda bir derýa bar şol tarapa ýöräberdim. Şäher onsuzam beýlekilerden tapawutly, kän milli geziljek ýa-da taryhy gezilip bilinjek ýerler ýok. Senagat şäheri bolandygy belli bolyp dur.

Azajyk aýlananymdan soň derrew telefonyň karta (maps) programmasyndan Google ofisleriniň bolan ýerini öwrenip şol binaň ýolyny tutdym. Ýolda çynym ýüzlerçe gezek aşyk boldym (:D). Google ofislerine baryp “Içerini gezsem bolarmy?” diýip soradym ýöne rugsat bermediler. Şedýdip ýene töweregi aýlanyp yzyma gaýtdym, agşam sagat ondaky Viýenna gidýän otlydan otyrgyç book edip garaşmak bilen bolyberdim.

Şwisariýa gelip iň gowy edip bolaýjak zatlaryň biri lyžada taýmak ýa-da snowboarding etmek. Alp daglar ýanyňda we meniň arkamda uly sumka bar bolsa daş töwerekdäki köp kişide lyža we snowboarding bardy. Nesip bolsa başga bir wagt..

Wiýenna

Wiýenna ýolçylygynda compartment-de Çehiýa respublikasyndan bolan bir gyz hem bardy, Wiýenna we Prague hakynda birnäçe maglumatlar alyşdyrdym. Özüne hem Turkmenistan, milli zatlarymyz hakynda gürrüň berdim. Wagt derrew geçdi we irden Wiýenna wokzalyna geldim.

Şu wokzal iň halan wokzalymyň biri, mugt internet we kompýuteri ulanmak üçin ýörite tokly ýerler bardy. Ýöne edil Zürich ýaly gymmadyrak şäher, çaý alanymdan bildim, iki ýewro töweregidi.

Howa sowygrak bolansoň kän daşaryk çykasym gelmedi, wokzalda otyryp blog ýazmak, internetde bolmak has gowy gelýärdi. Ýöne 1-2 sagat daşaryk aýlanyp gaýtdym. Wiýenna Mozart ýaly artistleriň dörän şäheri ekeni, suwinirlerden we Mozart şekilli şokoladlardan belli. Ýöne bir metro duragynda, üýtgeşik hajathana duş geldim, hajathanada klasik, opera aýdymlar çalýardy. (:D) Girip görmedim ýöne işiňi saz bilen bitirmek göz öňüme geldi..

Wiýenna hem köp aşyk bolan şäherlerimiň biri.. Derrew aýlanyşdyryp Prague ýolyny tutmak bilen boldym.

Prague

Şäheri daň bilen sagat bäş töwerekleri geldim. Ilkinji üns beren zadym Çek sözlerine düşünmegim, bular hem öň Sowýet baknalygynda bolany üçin köp zat birhili tanyş geldi. Hatta aýdymlary hem bir ors aýdymlaryna meňzeýädi. Orsça düşünýän hem köp, ýöne otlydaky sohbetdeş bolanym “Kän orsça gürleme, halanoklar” diýipdi. Menem köp ýerde iňlis dilinde düşünişdim.

Bu ýerde hem howa gaty sowykdy, gün çykýança wokzalda garaşdym. Soň derrew bir turist magazinden bir karta alyp gezmäne başladym. Prague-da gaty köp taryhy ýerler bar ekeni. Ol tower bul tower, Kurlev koprusi wş ýerleri aýlanyşdyrdym. Howanyň sowyk bolmagy hem özüme hem gezelençe täsir etmäne başlady, elim ellikli bolsa hem barmaklarym doňýady käwagt. Günortan töwerekleri gün çykyp azajygam bolsa daşaryny ýylatdy bolmasa gaty sowykdy. Derrew şäheri gezişdirip öýlän sagat dörtde Berline gitýän otlydan ýer aýyrtdym.

Italiýada Birnäçe Şäher

2012-01-29T00:00:00+00:00

2012 ýylyň 29-njy ýanwary - Milan-Zurich arasyndaky otlyda

Pisa

Gije ýarysyna sähel galanda, Livorno portyna geldik, derrew gämiden düşüp taksi tutyp wokzalyň ýolyny tutdym. Özüm ilki ýörärn wokzala çenli diýýädim ýöne soraşdyrsam gaty uzak ekeni.. Wokzaldaky gümrük işgärlerine haýyşt edipdim taksi çagyryp beräýiň diýip. Ýolda taksist bilen çat pat gürrüňdeş boldym, asyl bugün wokzal işgärleri “strike” (grev) edipdirler, haý bolmady bula diýip wokzala gelsem, ýörite bilet alynýan maşynlar bar ekeni, gelenimden edil on minut soňam bir otly Pisa-a gidýän ekeni, seretsem ýerli otly (Eurail biletlileriň ýerli gatnawlar üçin bilet almaklary hökman däl.) derrew liniýa baryp garaşdym.

On bäş minut soň Pisa wokzalyna getirdi, gidip ýakynda bir otelden otag tutdym (30 euro). Ertir sagat on bir töwerekleri şäheri aýlanmana başladym, Pisa şäheriň ýerleşiş düzgünini gaty haladym (aslynda Italiýada gezen hemme şähiriňkini haladym) edil wokzaldan garşyňa çykan ilkinji ýol bilen göni gidiberseň öňüňden ýap çykýar, köprini geçip ýene göni dowam ediberseň, dünýäniň täsinlikleriniň biri bolan Pisa binasyna eltýär..

Şäher kän uly däl ýöne gaty owadan, sada .. Aýlanyp, surat çekişdirdim, ýeke gezmek hezilem welin iki ýa-da ondan köp bolsaň has gowy, üýtgeşik suratlar çekse bolardy :D, herkim Pisa binasyny itekläp ýa-da ýykalmagyna päsgelmiş ýaly duryp surata düşýär. Herhili düşýän bardy özä, depýäni haýsy, barmagy bilen saklaýany haýsy wş. Men binaň ýokarsyna türkmen baýdagy asyp aşakdan hem biri çekse bet bolardy diýen fantaziýa kelläme geldi ýöne ýeke özümdim, ýokarda şol wagtda aşakda hem tanyş gerekdi .. Özümem hany itekläp düşeýin surata diýip geçip barýan iki sany gyza meň suratymy çekäýiň diýip haýyşt etdim, olar bilen azajyk sohbetdeş bolamdan soň, nirelidigimi soradylar, Türkmenistan diýseň herkim biläýenok soň nirede ýerleşýär, Kazagystan, Özbegistan goňşy döwletler wş diýişdirip düşündirýäň. Olaram “Ýöne seň iňliçäň gaty gowy!” (But you speak english quite good!) diýenlerinde monça boldym. Köp kişi biziň ýurdymyzy bilenok, şotaýlardan gelen adamlaryň özleri bilen sohbetdeş bolmagyna haýran galýalar..

Ýadap agşam sagat on töwerekleri wokzala geldim, Roma gitmek maksady bilen otly wagtlaryna seretsem irden bäş töwerekleri biri bar ekeni. Şol wagta çenli wokzalda garaşdan, sagat 12 çenli McDonalds-da ondan soňam sowyk, bomşlaryň üýşýän ýeri bolan wokzalda .. Menem bir burçy eýeledim we kitap okap aýdym diňläp wagty geçirdim ..

Rim

Bäşde ýola çykyp sagat on töwerekleri Rim (Roma) şäher wokzalyna geldim. Wokzal gaty uly ekeni ilkinji üns beren zadym, 30 töweregi otly liniýasy bardy. Derrew daşyna çykyp ilkinji adama “Kolossal” haýsy tarapda diýip soradym, salgy beren tarapyna tarap ýöredim. Şol wagtda jaýlara üns berdim, aý söz ýoklaý, çynym biz wagtynda palçykdan galalar gurýakak bular eýýäm daşdan jaýlar, mermer monument, statue wş gurypdyrlar.

Kolossala gelemde onsuzam haýran galdym .. Wi birem, Kolossal, hol bara Gladiýatorlaryň uryşýan ýeri şonyň galyndysy (surat goýmakçy). Daş töweregine aýlanyşdyryp ýene bet bina bolan, Sezaryň jaýy diýdilermi şonýaly bir bet mermer bina gurypdyrlar. Ony hem aýlanyp üstüne çykyp şäheri synlap yza gaýtmak bilen boldym. Sebäbi belli bir wagtdan soň aýak diýen edenok, birem şugün otlyda zordan 3 sagat töweregi uky alyp bilipdim.

Wokzala gelip, altyň ýarynda bir otly Florence (Florentina) gidýärdi şoňa mündim we ýatmak bilen boldym.

Florentina

Gelemde eýýäm sagat agşam dokuz töwerekleri. Iki sany Hostel adresi alypdym internetden ikisine hem gitdim (telefonyn maps app-i sag bolsyn). Ýöne ikisinde hem hiçkim gapyny açmady. Borlaý diýip şäheri gezmelemek bilen boldym.

Çynym bärsi beýlekilere görä has hem betdi birhili, bugün altynjy gün bolandanmy nämemi herkim daşarda ýalydy, ýaşy garrysy, owadanja gyzlar, oglanlara üns bermändirin, herkim ýolda .. Şäheriň özi hasam. Bärde hiç surata düşürmedim (gije bolandanmy kän elim fotoaparatly gezmedim. Şäheriň binalary uly uly we dykma dykma ýagny köçelerden galan ýerler diňe owadan we uly bina. Özümä şu ýerde okardym ýa-da ýaşardym ..

Soň ýene yzyma wokzala gaýtdym, gije dörtde Milana gitýän otla münmekçidim. Sagat ýaňy bir ikiň ýary töwerekleri ýene garaşmak üçin wokzalyň iki ädim ýanyndaky McDonalds-a gitdim.

Milan

Irden sagat dokuz töwerekleri Milan merkez wokzalyna geldim. Derrew internetden bir iki sany hostel adresi alyp metro bilen iň ýakyna gitdim. Baran hostelimde boş ýer ýok ekeni ýöne maňa ýakynragynda bolan bir hosteli salgy berdi, belki olarda bardyr boş ýer diýip. Tapan hostelim, California Hostel, olarda hem edil şol wagt boş ýer ýok ekeni, ýöne sagat öýlän üçden soň boş ýer bolar diýdiler. Sumkany goýsam bolarmy diýip, San Siro nähili gitmeli salgysyny alyp aýlanmana gitdim.

Stadion şäheriň daşragynda ýerleşýän ekeni, taş töweregini aýlandym, birnäçe adamlar bilen gürrüňdeş bolsam asyl agşam AC Milan - Caiglari oýyny bar ekeni, bilet bahasyny sorasam iň arzany 20euro (iň sonky, 5-nji setir), gidip 22euro 3-nji setirden bilet aldym. Ýöne ýanymda ýekeje hem Milana degişli zat ýokdy, AC Milan magazynyna barsam bazar güni sebäpli ýapyk ekeni. Ýakynlarda kän futbol kluba degişli odyk budyk satýan bardy şolardan bir klubyň şarfyny satyn aldym. Şeýdip birden hemme zat öz özünden bolyberdi, agşamky oýuna biletim bardy birem ýer hem boşaýsa hostelde diýip şäheriň merkezine gitdim metro bilen.

Milan şäher merkezinde meşhur bir Katedral (church-yň ulusy) bar, üýtgeşik arhitekturasy bar, Gothic Catedral diýýäler. Otada butaýda aýlanyp yzyma hostele gaýtdym.

Agşam stadiona oýyn görmäne gitdim, aslynda ilkinji gezek stadiona bir futbol oýyny görmäne gitýänimi aňdym, ýerimi tapyp otyrdym. Bilmedim gowy balet etdimmikän diýýän özä (:D), her golda türkmen baýdagy açan boldym, wş.. Şol ýerde hem ýanymda otyran bir iki sany ýerli janköýerler, “Sen nireden, niräň baýdagy bul?” diýip soradylar, menem “Turkmenistandan” diýip soňam ilkinji eşiden ýurtlary bolansoň, nirede ýerleşýänini, goňşy ýurtlary aýdyşdyrdym. “Şotaýdan AC Milana balet etmäne geldiňmi?” diýip sorady, menem “Elbette!” diýdim, aslynda aldadym, oýyny görmegim hiçhili planlanmandy. Aý Milan janköýerleri begensin diýdimdä..

Hostelde hem boş ýer bar ekeni, iki günlik ýer aldym, toplamda 30euro. Bir gün gowy dynç almagy planlaýadym. Otaýda hem 3 sany ors oglan bilen tanyşdym, fençing (fihtawaniýa) ýaryşa gelipdirler, Kyrgyzystan adyna ýaryşýan ekenler. Olar maňa rusça men olara iňlisçe, rusça garyşyk dilde gürleşipdim. (Çynym orsçamy gowylaşdyrmy!) Hostel kuhnýasynda yzakly gün gülüşip otyrdyk, soň bir ýerlerden ýaponlar geldi olar bilen goşylyp bir gün gowy dynç aldym.

Arada bir wagt tapyp San Siro AC Milan store hem aýlanyp gaýdypdym. Magazindan çykan ýanyňda klub muzeýi bardy. Göreýin diýip barsam eýýäm ýapjak bolyp durdylar öýlän bäş töwerekleridi, gijä galypdym.

Ertesi gün, 31-nji Ýanwar, irden wokzala gaýtdym. Sagat ýedide Zurich gitýän otla bilet alyp ýola çykdym. Şuwagt hem bulary otlyň içinden ýazyp otyrn..

Barcelona We Başdan Geçirenlerim

2012-01-26T00:00:00+00:00

2012 ýylyň 26-njy ýanwary - Orta ýer deňizinde bir ýerler

Şäher Barada

Lissabondan gije ýola çykyp, Madrid-den transfer edip, soňynda günortan töwerekleri Barcelona şäherine geldim, sag aman. Güneşli we salkyn howa bardy. Derrew wokzaldan çykyp daş töwerege aýlanmana gitdim. Bir ýolyň soňynda bir aýlawa we aýlawyň ýokarsynda depede hem betje bina göze ildi, şol tarapa ýöredim. Şäheriň onsuzam daş töweregi uly depeler bilen gabalan ýalňyşmasam, her tarapda ýaşyl depeler bardy. Depedäki ýer bir milli muzeý ekeni, daş töweregi turistik ýerler. Owadan ýerler surata düşürişdirip bir iki sagat aýlandym.

Pulymyň Ogurlanmagy

Boldy indi diýip yzyma barýarkam, öňümden 4-5 sany çaga (14-15 ýaşly) gyz çykyp, “Biz maýyp çagalar üçin ýardym toplaýas, golyňyz we şaheriňizi ýazyp 5 euro beräýiň?” diýdiler, menem borlaý diýip, gapjykdan puly çykaryp berdim, onýança olaram ID-ňizi hem görkeziň diýişdirip gapjyga ýapyşyp kimlik kartymy aljak boldylar, bolýar ýok diýip ýaňy aýrylyp 1 minut ýöremänkäm, kellä depdi, şulara bir oýuna oýnady diýip gapjyga seretsem beýleki jübüsindäki 4 sany ellilik ýok! Derrew ylgap baryp biriniň elinden tutyp “Bolyň puly beriň bolmasa polisiýa aýtýan häzir!” diýip, bir bir ýarym minut gygyryşdyk, soň tutan gyzym kurtkasyny çykardy we hemmesi birden gaçyp gitdiler. Yzlaryndan ylgajagam boldym welin arkadaky 7-8 kg bilen ylgap bilmedim. Elimde kiçi gara kurtgadan başga zat galmady. Derrew ýanymdaky dükana baryp polise jaň edäýiň diýdim, olar eýýäm eşidip, görüp jaň edipdirler. 3-4 minut soň polis geldi we soraşdyryp, eger report etmek isleseň merkeze gelmeli bolarsyň diýdiler menem ylalaşyp gitdim. Aňyrda hem kinolardaky ýaly (:D) birnäçe çaga kriminallaryň suratlaryny görkezdiler, aklymda galan we meňzeýänlerini görkezişdirdim, hemme zady aýdyşdyrdym, report ýazyp özüme hem bir kopiýasyny berdiler we aýryldym..

Ýola çykmankam maňa iň ýakyn bolan adamlaryň biri, sadaka bermäni ýatdan çykarma diýipdi. Menem sadaka berer ýaly musulman barmy töwerekde bahanasy bilen bermändim, olam niýetiň sadaka bolsa bolýar diýipdi.. ogurlykdan soň ilkinji kelläme gelen zatdy şu. Haýyrlysy, ýol we başymyň sadakasy bolsyn diýip ýenede gezelençden lezzet almak bilen bolyberdim. (aslynda özümide gaharym gelýar, eldäki pullaryň hemmesini bir ýerde tutmak gaty howply, her ýerde birnäçe manat saklamak amatly we howpsyzrak.. haýyrlysy)

Indi Niräk

Ýene bir sagat ýaly şäheriň içinde ýöredim, gün ýaşypdy, soraşdyryp ýene Barcelona Sants wokzala geldim, derrew gidesim gelýärdi Italiýa ýöne otly ýok ertir agşama çenli, ýakynrakda bolan Fransiýadaky Marselle, Nice şaherlere hem otly ýokdy. Soň gämi (ferryboat) bilen gideýinle diýip metro bilen Barcelona port-a gitdim.

Derrew bilet soraşdyrdym, şugün gije ýarysy bir gämi Livorno-a (Italiýada Pisa şäherine ýakyn bolan şäher) gitýän ekeni, derrew biledi satyn aldym (19 sagatlyk Barcelona-dan Livorno-a gämi ýolçulugy Orta ýer deňizinden). Sagat entäk ýaňy bir on töweregi bolansoň birzatlar iýişdirdim we töweregi gezdim, gaty bet ýerler ekeni port töwerekleri, tüýs jübitler üçin (:D).

Onça ýadawlykdan soň öli ýaly ýatyp turyp, şu wagt gämiden Orta ýer deňiziň bir ýerlerinden ýazyp otyrn şulary. (şu wagt internet ýok, şoň üçin bolan wagty git push etmeli boljak..)

Eurail Bilen Gezelenç

2012-01-24T00:00:00+00:00

2012 ýylyň 24-nji ýanwary - Lissabon

Giriş

Köpüňiz öň hem eşidensiňiz, Interrail ýa-da otly bilen ýewropa ýurtlaryny gezmek, Eurail bolsa ýewropadan başga ýerde ýaşaýanlar üçin şu mümkinçiligi ýetirýär. Eurail.com-dan islän bilet görnüşini saýlap gezelenç edip bilersiňiz. Bilet görnüşleri,

Global Pass — ýagny ýewropaň islendik ýurtlaryny 15 gün içinde gezmek
Regional Pass — birnäçe gün ýewropada 2-3 döwleti saýlap gezmek
Flexi Pass — alty aý içinde islendik gün, toplamda 15 gün islendik ýeri gezmek

ýaly bilet görnüşleri bar.

Biletler online satyn alynýar, we beren adresiňize iki hepde içinde ýetirilýär. Soň öňümizdäki alty aý içinde islendik wagt gezelençe başlap bolýar, ondan soň bilet köýýär. Şeýle sorag ýüze çykyp biler, “Wiza gerekmi?” gezelenç üçin, meň özümä gerek däl diýip bilýän, biletiň özi wiza ýerine geçýärmikä diýýän, haýsam bolsa bir ýewropa döwletine gelip başlabermeli aýlan-çaýlaňa. Ýöne anyk bilmeli, öňümizdäki günlerde soramakçy. Islän soragyňyzy Eurail Facebook sahypalarynda birem hepde içi chat wagtlary bar hergün 5 sagatmy birzat, bolmasa hem mail bilen sorap bilersiňiz, iki iş güni içinde jogap berýärler. (sales@eurail.com)

Taýýarlyk

Gezelenje taýýarlyk barada ýüzlerçe maslahat bar, ýöne şu ýerde öz eden zatlarymy ýazmakçy. Ilki bilen rugzaga (sumka) nämeler alyşdyrdym,

Eşikler: gyşlyk geýip bolaýjak, şarf, şapka wş.
Ýuwynmak üçin gerek zatlar, sabyn, şampun, diş pastasy, iň esasy hem sakgal syrýan maşynka :D
Kitap: Game of Thrones (A Song of Ice and Fire, Book One), nesip bolsa gutaryp bilerin ýolda okap
Music: AC/DC butin albomlaryny download etdim, hemme aýdymlaryny diňlemekçi we birem Pink Floyd-dan hem öň diňlemedik albomlarymy aldym
Laptop: men munsyz gezip bilmeýän, iň agyram şul
Fotoaparat
Pasaport we Bilet
We başgada birgiden akyr-ukur, karta, şarj enjamlar, kompass, fonarik, first aid üçin ýara bagy wş

Bulardan başga hem telefonyma Bahn (Germaniýaň otly sistemasy) app gurdym, ýewropadaky hemme şäherler arasy otly wagtlaryny tapsa bolýar, ýenede Free Wi-Fi Finder app we birnäçe şäher guide-leri bolan Lonely Planet app-lerini guryşdyrdym.

Kyn, howply we şol bir wagtda gyzykly on bäş gün garaşýar..

Tourists don’t know where they’ve been, travellers don’t know where they’re going.

SOPA Barada Gysgaça

2012-01-18T00:00:00+00:00

2012-njy ýylyň 18-nji ýanwary - Lissabon

Ýaňy pikirlerde Wikipedia iňlis dilinde näme üçin ýapylýar wş soraglar soraldy. Şu mowzukda azajygam bolsa düşündirip bilişimçe sebäbini ýetirmekçi. Her bir internet ulanyjysynyň bunuň sebäbi barada maglumat edinmegi möhüm!

SOPA (Stop Online Piracy Act) eýýäm näçe aý bäri Amerikada kanun boljak bolyp ýör. Gysgaça aýtsak SOPA copyright haklary goramak üçin kanunlaşdyryljak bolýar. Hä bize näme dahyly bar diýip bilersiňiz.

Eger SOPA kanun bolyp kabul ediläýse Amerikada, hakkyny gözleýän USA raýatlary congress kanuny bozan saýty barada dawa açandan soň şol saýty gözleg motorlary (Google, Yahoo, Bing) tapmagy (indexing) gadagan, online reklam bilen işleşýänleriň iş alyşmagy gadagan we iň möhümi ISP-leriň (internet service provider) şol saýta girilmegini gadaganlamagyna getirip bilýär.

Asyl SOPA-nyň maksady hem şol, ýagny USA daşynda bolan kino wş zatlary paýlaşýan saýtlary bloklamak wş. Kanun bolaýsa köp ulanýan saýtlarymyzyň ýapylmagy diýmek (torrent wş).

Bu kanuna, Google, Facebook, Twitter, Zynga, eBay, Mozilla, Yahoo we LinkedIn açyk şekilde garşy bolandyklaryny yglan edip birnäçe USA congressman-a hat ýazypdylar..

Eger kanun giriziläýse, ilki bilen Youtube-a täsiri uly we Vimeo, Flickr ýaly saýtlaryň bolsa ýapylmagy diýmek.

Gysgaça şeýleräk.

CS Papers - BitTorrent

2012-01-18T00:00:00+00:00

18 January 2012 - Lisbon

1 Introduction

Original paper is here. As you know BitTorrent is Peer-to-Peer file sharing protocol.

2 Technical Framework

2.1 Publishing Content

The .torrent file contains information about the file, its length, name, hashing information and url of a tracker. Trackers are responsible for helping downloaders to find each other. A downloader sends information about what file it is downloading, what port it’s listening on, and tracker responds with a list of contact information for peers which are downloading the same file. Downloaders use this information to find and connect to each other. To make a file available, a ‘downloader’ which happens to have the complete file, known as seed, must be started.

2.2 Peer Distribution

The tracker’s responsibilities are strictly limited to helping peers to find each other. All logistical problems of file downloading are handled in the interactions between peers. In order to keep track of which peers have what, BitTorrent cuts files into pieces of fixed size. Each downloader reports to all of its peers what pieces it has. To verify data integrity, SHA1 hashes of all pieces are included in the .torrent file, and peers don’t report that they have a piece until they’ve checked the hash. Peers continuously download pieces from all peers which they can.

2.3 Pipelining

BitTorrent facilitates the pipelining by breaking pieces further into sub-pieces over the wire, typically sixteen kilobytes in size, and always keeping some number, typically five, requests pipelined at once. This will avoid a delay between pieces being sent, which is disastrous for transfer rates. The number for pipelining can be selected as a value that will reliably saturate most connections.

2.4 Piece Selection

Selecting pieces to download in a good order is very important for good performance.

2.4.1 Strict Priority

BitTorrent’s first policy for piece selection is that once a single sub-piece has been requested, the remaining sub-pieces from that particular piece are requested before sub-pieces from any other piece. This will be good for getting complete pieces as quickly as possible.

2.4.2 Rarest First

When selecting which piece to start downloading next, peers generally download pieces which the fewest of their own peers have first, a technique referred as “rarest first”. This technique does a good job of making sure that peers have pieces which all of their peers want, so uploading can be done when wanted. It also makes sure that pieces which are more common are left for later, so the likelihood that a peer which currently is offering upload will later not have anything of interest is reduced.

2.4.3 Random First Piece

An exception to rarest first is when downloading starts. At that time, the peer has nothing to upload, so it’s important to get a complete piece as quickly as possible. Rare pieces are generally present on one peer, so they would be downloaded slower than pieces which are present on multiple peers for which it is possible to download sub-pieces from different places. Until the first complete piece is assembled, pieces to download are selected at random and then strategy changes to rarest first.

2.4.4 Endgame Mode

Closer to the end of download, a peer with very slow transfer rates may delay download’s finish. To keep that from happening, once all sub-pieces which peer doesn’t have are actively being requested it sends requests for all sub-pieces to all peers. Cancels are sent for sub-pieces which arrive to keep to much bandwidth from being wasted on redundant sends.

3 Choking Algorithms

To cooperate peers upload, and to not cooperate they ‘choke’ peers. Choking is a temporary refusal to upload; it stops uploading but downloading can still happen and the connection doesn’t need to be renegotiated when choking stops. A good choking algorithm should utilize all available resources, provide reasonably consistent download rates for everyone, and be somewhat resistant to peers only downloading and not uploading.

3.2 BitTorrent’s Choking Algortihm

Each BitTorrent peer always unchokes a fixed number of other peers (default is four), so the issue becomes which peers to unchoke. Decisions as to which peers to unchoke are based strictly on current download rate. Calculating current download rate meaningfully is a surprisingly difficult problem; the current implementation essentially uses a rolling 20-second average. BitTorrent peers recalculate who they want to choke once every ten seconds, and then leave the situation as is until the next ten seconds period is up.

3.3 Optimistic Unchoking

Simply uploading to the peers which provide the best download rate would suffer from having no method of discovering if currently unused connections are better than the ones being used. To fix this, at all times a BitTorrent peer has a single ‘optimistic unchoke’ which is unchoked regardless of the current download rate from it. Which peer is the optimistic unchoke is rotated every third rechoke period (30 seconds).

3.4 Anti-snubbing

Occasionally a BitTorrent peer will be choked by all peers which it was formerly downloading from. In such cases it will usually continue to get poor download rates until the optimistic unchoke finds better peers. Therefore, if over a minute goes by without getting a single piece from a particular peer, BitTorrent assumes it is ‘snubbed’ by that peer and doesn’t upload to it except as an optimistic unchoke.

3.5 Upload Only

Once a peer is done downloading, it no longer has useful download rates to decide which peers to upload to. The current implementation then switches to preferring peers which it has better upload rates to, which does a decent job of utilizing all available upload capacity and preferring peers which no one else happens to be uploading to at the moment.

EMDC Courses

2012-01-15T00:00:00+00:00

15 January 2012 - Lisbon

Introduction

On the summer of 2011 after graduating from METU Computer Engineering, I have been accepted to European Masters in Distributed Computing (EMDC) joint Erasmus Mundus programme between KTH Royal Institute of Technology (KTH) in Sweden, Instituto Superior Técnico in Portugal (IST) and Universitat Politècnica de Catalunya (UPC) in Spain. It is a two-year Master’s programme including compulsory mobility for the students.

I have been accepted to IST - KTH track, which means I will be studying in the Instituto Superior Técnico for two semesters and another semester in KTH Royal Institute of Technology and for final semester will be doing my thesis in Institute Superior Técnico again.

After long efforts and procedures of getting visa, I have managed to get to the Lisbon and adapt to the beautiful city, to the university and meet with great friends.

This semester (Fall 2011) at IST I have taken four master’s courses,

Parallel and Distributed Computing
Peer-to-Peer Systems and Overlay Networks
Cloud Computing
Network and Computer Security

All of these courses were challenging and interesting, I have learned whole new technologies, concepts and had hands on practice doing projects.

Parallel and Distributed Computing

This course was most familiar for me since I had Parallel computing course while studying my Bachelor’s degree. Nevertheless I have learned new things about distributed computing; distributed architecture, OpenMP, MPI, designing and implementing distributed and parallel algorithms and analyzing them using several metrics.

Course project was to implement network slack computation both using OpenMP (shared memory) and MPI (distributed memory).

Peer-to-Peer Systems and Overlay Networks

This was one of the interesting courses. Starting from BitTorrent, eMule and then learning about unstructured p2p systems, Gnutella; then structures ones, Chord, Pastry, Kademia and Can. After that were a little bit challenging, we have studied Gossip, One-hop protocols, distance estimation (Vivaldi algorithm), content distribution (Akamai), load balancing in p2p systems, etc.

Course project was to design Http p2p proxy.

Cloud Computing

This course was also very interesting. Starting from clusters, grids and eventually about cloud technologies. Implementing mini MapReduce using Hadoop, cloud technologies Amazon AWS, Google App Engine and MS Azure.

Course project was to design and implement mini web indexing system that will be using all cloud systems. We, project team, had implemented page-rank algorithm on Amazon EC2, stored data on MS Azure database and implemented web front-end on Google App Engine communicating with Azure database. Trying and learning these was very interesting.

Network and Computer Security

This course was my least interesting. We have learned lots of protocols and key encryption algorithms, both symmetrical and asymmetrical. Moreover, about the network certification organizations.

Since semester is over and we had first round of exams (hope we pass them) we can concentrate to the next learning adventures.

Miradouro da Graça

2012-01-10T00:00:00+00:00

2012-njy ýylyň 10-njy ýanwary - Lissabon

Lissabonda howa köplenç maýyl, gyşyň ortasynda gezip ýörs ýöne entäk ne gar bar nede sowyk howa. Käwagtlar ýagyş ýagýar diýäýmeseň howa köplenç güneşli we maýyl.

Şugün dostlarymyň biri, ýakynymyzda çaýlamak ýa-da kitap wş okamak üçin gowy bir ýer bar diýdi we biz ekzamine taýýarlanmak üçin gitdik şol ýere. Bilýän gülýäňiz ýöne öýde oturyp okamak ýerine şol ýere gidenime begenýän, azajygam bolsa kelläň dynç alýar, we öz komfort zonaňdan çykýaňda..

Mirodouro da Graça, terjime etsek Graça viewpoint-y (ýerliler miradouro diýýär) ýagny şäheriň bir bölegini synlap biljegiňiz ýer diýmek. Lissabonyň duran ýerinde köp beýikli pesli ýerler bar we şäheri synlamak üçin birnäçe depe bar. Bularyň biri hem Mirodouro da Graça, turistleriň, studentleriň we başgada kän adamlaryň dynç almak üçin ugraýan ýeri. Çaý, tost iýmek içmek üçin hem kiçijek kafe bar..

Şondan soň birnäçe gezek gitdim şol ýere (ekzaminlere taýýarlanmak üçin :D) ..