As promised, here is the solution to the problem that I found. Though I'll note that this is a little involved. In order to make this work you will need to be create a Java job in TOS and have another package on your system that allows you to write and compile java. On my system I'm using netbeans working with that latest JDK.
1) Create the job in TOS.
2) If you are outputting to a MySQL component, make sure that in the advanced settings of the component you have the following string as part of your optional JDBC parameter.
&characterEncoding=UTF-8
In my job, my optional JDBC parameters look like this....
"noDatetimeStringSync=true&characterEncoding=UTF-8"
I have not been able to test this to my satisfaction as of yet, but my server has a default client character set setting of UTF-8, so I don't know why I have to specify this as an optional JDBC parameter, but I do know that regardless of the system server variable being set, if I don't have this optional JDBC parameter in there it doesn't work.
3) Run the job to make sure there are no logic errors. Essentially you are just checking to make sure that the job is moving data from point A to point B. At this point if it's not moving non-latin characters it's not an issue, you just want to make sure that the only problem with your job is the inability to move non-latin characters.
4) Export the job. In your repository, under Job design, right click on your job and select ?Export Job?. When you do the job export MAKE SURE you have ?Source Files? checked as this is what you are going to need.
5) The job typically exports in a zip compressed format, though you can choose to decompress this as the job is exported. Regardless, if you haven't decompressed the job file, do so now.
6) Now find the file that has the source code for the job that you just created. You'll find it in a path SIMILAR to this in the decompressed data....
\<job name>\<job name>\src\<project name>\<job name>\<job name>.java.
So for example, if my job name was ?transfer?, version 0.1, and was part of a project called ?Company?, and the job's files were extracted to c:\temp, you would find the file here....
C:\Temp\transfer_0.1\transfer\src\Company\transfer_0_1\transfer.java.
Find this file and open it in your java development studio of choice.
7) Look for the readString function. In my code it looks like this....
private String readString(ObjectInputStream dis) throws IOException {
String strReturn = null;
int length = 0;
length = dis.readInt();
if (length == -1) {
strReturn = null;
} else {
if (length > commonByteArray.length) {
if (length < 1024 && commonByteArray.length == 0) {
commonByteArray = new byte;
} else {
commonByteArray = new byte;
}
}
dis.readFully(commonByteArray, 0, length);
strReturn = new String(commonByteArray, 0, length, utf8Charset);
}
return strReturn;
}
delete this and replace it with this....
private String readString(ObjectInputStream dis) throws IOException {
String strReturn = null;
int length = 0;
length = dis.readInt();
if (length == -1) {
strReturn = null;
} else {
strReturn = dis.readUTF();
/*
if (length > commonByteArray.length) {
if (length < 1024 && commonByteArray.length == 0) {
commonByteArray = new byte;
} else {
commonByteArray = new byte;
}
}
dis.readFully(commonByteArray, 0, length);
strReturn = new String(commonByteArray, 0, length, utf8Charset);
*/
}
return strReturn;
}
The biggest thing that we're changing here is just commenting out some un-needed code.
Bare in mind that this function will occur several times in the code. I believe there is at LEAST 1 for every input component that you have in your job, so make sure you get ALL of them.
😎 Last, there is a try catch for getting context. You should be able to find this by searching for the following string in your code....
// call job/subjob with an existing context, like:
// --context=production. if without this parameter, there will use
// the default context instead.
Just below it you will find the following code....
java.io.InputStream inContext = stateprov.class.getClassLoader()
.getResourceAsStream(
"ideallife/stateprov_0_1/contexts/" + contextStr
+ ".properties");
This loads data from the context properties file that is exported along with the job. What we want to do is change this to just point it directly to the file instead of having the code try to figure out where the file is. Going back to the example that we used earlier when finding the java source code, you will be able to find the file in the following location....
C:\Temp\transfer_0.1\transfer\Company\transfer_0_1\contexts\Default.properties
So in this case we would want to revise the code to look like this....
java.io.InputStream inContext = new java.io.FileInputStream("C:\\Temp\\transfer_0.1\\transfer\\Company\\transfer_0_1\\contexts\\Default.properties");
//stateprov.class.getClassLoader()
// .getResourceAsStream(
// "ideallife/stateprov_0_1/contexts/" + contextStr
// + ".properties");
What we've done here is set the inContext to look to the file. We use double backslashes because the backslash is an escape character and needs to be noted twice to be handled as a litteral backslash. The four lines that followed have been commented out as they are no longer needed.
One you have made the revisions, compile and run the code. You will find it will run EXACTLY like the job you made using the GUI, only now it will actually transfer non-latin characters.
And boom goes the dynamite.
- Peter
PS. Devs are complete prima donnas, so I have to give credit to the dev that found the work around. His name is Shlomi Rosenzweig and he rocks. Unfortunately there is not enough space on this thread page for his ego, so this will have to suffice.