tSchemaComplianceCheck not validating correctly

One Star

tSchemaComplianceCheck not validating correctly

I set up a compliance check to report errors for null fields. However, it ignores these rules but when I have a character field with nullable cheched, it rejects null fields.
Community Manager

Re: tSchemaComplianceCheck not validating correctly

Hello

Which version of TOS do you use? I check it with TOS3.1.0 and it works fine. Please see my screenshots.

Best regards

shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: tSchemaComplianceCheck not validating correctly

I was using 3.1.1 r24830. I'll try it in 3.1.0 to see if it is a bug.
I tried the validation in 3.1.0 and I get the same result.

Looking at the generated code it seems wrong. There is no null checking for Gender and ProgramCode does not have an == null check.

if (row1.Forenames == null) {

ifPassed_tSchemaComplianceCheck_1 = false;
errorCode_tSchemaComplianceCheck_1 += 4;
errorMessage_tSchemaComplianceCheck_1 += "|empty or null";
}
if (row1.Forenames != null) {

if (row1.Forenames.length() > 150)
row1.Forenames = row1.Forenames.substring(0, 150);
}

if (errorCode_tSchemaComplianceCheck_1 > 0) {

if (resultErrorCode_tSchemaComplianceCheck_1 > 0) {
resultErrorCode_tSchemaComplianceCheck_1 = 16;
} else {
resultErrorCode_tSchemaComplianceCheck_1 = errorCode_tSchemaComplianceCheck_1;
}
errorCode_tSchemaComplianceCheck_1 = 0;
}

if (errorMessage_tSchemaComplianceCheck_1.length() > 0) {
if (resultErrorMessage_tSchemaComplianceCheck_1
.length() > 0) {
resultErrorMessage_tSchemaComplianceCheck_1 += ";"
+ errorMessage_tSchemaComplianceCheck_1
.replaceFirst("\\|", "Forenames:");
} else {
resultErrorMessage_tSchemaComplianceCheck_1 = errorMessage_tSchemaComplianceCheck_1
.replaceFirst("\\|", "Forenames:");
}
errorMessage_tSchemaComplianceCheck_1 = "";
}

if (errorCode_tSchemaComplianceCheck_1 > 0) {

if (resultErrorCode_tSchemaComplianceCheck_1 > 0) {
resultErrorCode_tSchemaComplianceCheck_1 = 16;
} else {
resultErrorCode_tSchemaComplianceCheck_1 = errorCode_tSchemaComplianceCheck_1;
}
errorCode_tSchemaComplianceCheck_1 = 0;
}

if (errorMessage_tSchemaComplianceCheck_1.length() > 0) {
if (resultErrorMessage_tSchemaComplianceCheck_1
.length() > 0) {
resultErrorMessage_tSchemaComplianceCheck_1 += ";"
+ errorMessage_tSchemaComplianceCheck_1
.replaceFirst("\\|", "Gender:");
} else {
resultErrorMessage_tSchemaComplianceCheck_1 = errorMessage_tSchemaComplianceCheck_1
.replaceFirst("\\|", "Gender:");
}
errorMessage_tSchemaComplianceCheck_1 = "";
}
try {
if (row1.ProgrammeCode != null) {
String tester_tSchemaComplianceCheck_1 = String
.valueOf(row1.ProgrammeCode);
}
} catch (Exception e) {
ifPassed_tSchemaComplianceCheck_1 = false;
errorCode_tSchemaComplianceCheck_1 += 2;
errorMessage_tSchemaComplianceCheck_1 += "|wrong type";
}
if (row1.ProgrammeCode != null) {

if (row1.ProgrammeCode.length() > 50)
row1.ProgrammeCode = row1.ProgrammeCode.substring(
0, 50);
}

if (errorCode_tSchemaComplianceCheck_1 > 0) {

if (resultErrorCode_tSchemaComplianceCheck_1 > 0) {
resultErrorCode_tSchemaComplianceCheck_1 = 16;
} else {
resultErrorCode_tSchemaComplianceCheck_1 = errorCode_tSchemaComplianceCheck_1;
}
errorCode_tSchemaComplianceCheck_1 = 0;
}

if (errorMessage_tSchemaComplianceCheck_1.length() > 0) {
if (resultErrorMessage_tSchemaComplianceCheck_1
.length() > 0) {
resultErrorMessage_tSchemaComplianceCheck_1 += ";"
+ errorMessage_tSchemaComplianceCheck_1
.replaceFirst("\\|",
"ProgrammeCode:");
} else {
resultErrorMessage_tSchemaComplianceCheck_1 = errorMessage_tSchemaComplianceCheck_1
.replaceFirst("\\|", "ProgrammeCode:");
}
errorMessage_tSchemaComplianceCheck_1 = "";
}
try {
if (row1.ProgrammeTitle != null) {
String tester_tSchemaComplianceCheck_1 = String
.valueOf(row1.ProgrammeTitle);
}
} catch (Exception e) {
ifPassed_tSchemaComplianceCheck_1 = false;
errorCode_tSchemaComplianceCheck_1 += 2;
errorMessage_tSchemaComplianceCheck_1 += "|wrong type";
}
// validate nullable
if (row1.ProgrammeTitle == null) {

What I want to do is validate that 2 columns in the input file and reject them if they are null The columns are Gender character 1 with nullable unchecked in the file schema and ProgrammeCode string 50 with nullable unchecked in the file schema. How can I get the schema compliance check to trject records with null in these values?
I would also like to be able to apply rules to the validation of fields so that I can reject Gender if it is neither M or F. Can this be done?
Regards
Jan
One Star

Re: tSchemaComplianceCheck not validating correctly

Ok as well as creating incorrect Java by missing bits out, when I manage to get it to create almost correct code it produces the condition

if (row1.ProgrammeCode == null)
This doesn't work for empty strings in delimited files - need to test for 0 length

So this works ok

if (row1.ProgrammeCode.length() == 0 || row1.ProgrammeCode == null)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads