One more time I need to load a huge number of data in memory with a Perl hash. See previous post about this in When memory matters. This time the hash key is made of several values. The value corresponding to each key is an array of scalar data. I have 6 string values for the key and 10 numeric float values for the corresponding value.

I need to load a huge number of data in memory with a Perl hash. The value corresponding to each key is an array of scalar data. The key is most of the time created with a single field of my array, but it can be made of several fields. The number of fields in the array may vary a lot, but most of the time it will be around 5 scalar values.

New feature in Talend Open Studio 2.4, the ability to run iterations in parallel. After a tLoop or a tFileList you can set the "number of parallel executions" on the iterate link. If you're running on a quad core computer, it might be interesting to ask for 4 executions in parallel. 4 executions in parallel means that 4 "iterations" will be executed in parallel as long as some iterations are remaining. If your tFileList finds 1,000 files, you won't have 1,000 parallel executions, but 4, theoretically reducing the total execution time by 4.

With Talend Open Studio 2.4 and a Perl project, component tPipeRow appears in the palette. tPipeRow sends each input row to an external command, fetch the returned line and send it to the next component. tPipeRow does not launch the external script as many times as there are input rows, but only once at the data flow initialization. It makes the whole thing very performant. For each STDIN line, the external script must produce one STDOUT line, without buffering.

On a technical point of view, it's very interesting to see 2 scripts running in parallel and communicating through file descriptors. tPipeRow code is very short (but was not that short to write) because it simply uses IPC::Open2.

In Talend Open Studio, the multithreading option makes possible to execute 2 subjobs in parallel. It was implemented in Java code generation last summer (see feature 1335) for TOS 2.1 and I implemented it for Perl code generation last monday (see feature 3302). Current multithreading option was not implemented with threads in Perl, but with processes, I fork the parent process in children.

Talend Open Studio 2.3.0M2 is out. Let me list you what's new concerning Perl generation, compared to the current main release 2.2.3. As you will see, Perl code generation is still in progress :-) 13 new components, 8 new features in existing components. In this blog post, I only list news about Perl code generation, there are of course more new features, they are fully listed on the official ChangeLog page in releases 2.3.0M1 and 2.3.0M2.

In feature 2378, I've implemented MySQL specific extended insert mode. Extended insert means that instead of inserting lines one by one, you insert many lines in the same insert query. Don't get confuse with a transaction mecanism, it's not. The advantage is speed.

To illustrate the performance improvement we'll have in Talend Open Studio 2.3.0M2 using extended inserts, I've created a benchmark : we read lines from a delimited file and we insert them in a table. 3 simple fields per line (numeric id, firstname, lastname). 1 million of lines to insert.

We're using Debian Etch (with GNU/Linux) as a server at Talend office. We need to reach a remote Microsoft SQL Server database. The first step is to perform a select query in the command line.

We need to install FreeTDS: FreeTDS is a set of libraries for Unix and Linux that allows your programs to natively talk to Microsoft SQL Server and Sybase databases.. We have to define an "interface" for the Microsoft SQL Server in the FreeTDS "interfaces" file. At the end of the line,w use sqsh, a command line client for Sybase and Microsoft SQL Server.

Richard and I have both worked 2 weeks on a main improvement proposed by Richard.

.----------------------------------------------------.
| job        | TOS 2.1.1 | TOS 2.2.0M1 | improvement |
+------------+-----------+-------------+-------------+
| Scenario 2 |    20.8 s |      16.9 s |      18.8 % |
| Scenario 3 |    81.2 s |      30.4 s |      62.6 % |
'------------+-----------+-------------+-------------'

• Scenario 3 details in Talendforge wiki
• Scenario 2 details in Talendforge wiki

As a reminder for myself, here is a list of join examples using Oracle.