One Star

tSqoopImport and tSqoopExport do not have "custom" Hadoop Distribution

I would like to use Sqoop with our DataStax Enterprise version of Hadoop, but the Custom distribution seems to be missing on the "Sqoop" components, but the documentation says it should be there. Using the command line is not an option, as we invoke it in a non-standard way as well.
Also, we require some non-class things on the class path. I'm working internally to resolve this, and in the mean time I was able to use the tLoadLibrary component, and add them to the "Lib Path". That's a bit of a kluge, and I'm wondering if there's a better place to set these.
Also, we have a custom Pig UDF that requires a couple of environment variables. I tried the tSetEnv component, but to no avail. Is there a way to set them as JVM parameters, or in another logical location?
4 REPLIES
Moderator

Re: tSqoopImport and tSqoopExport do not have "custom" Hadoop Distribution

Hi,
For your issue 1, Sqoop components do not support the Custom distribution option, so far. See the online component reference tSqoopImport. It has been mentioned in distribution section.
For your issue 2 , I need make investigation and come back to you asap.
For your issue 3, Could you please give us your the screenshot of your component setting and work flow so that we can address your issue quickly.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Employee

Re: tSqoopImport and tSqoopExport do not have "custom" Hadoop Distribution

I've created a jira here: https://jira.talendforge.org/browse/TDI-26159
The custom in sqoop components will be included in 5.3.1
One Star

Re: tSqoopImport and tSqoopExport do not have "custom" Hadoop Distribution

For Issue #3, I'm not sure what you mean. I have a custom Pig Loader that is generally configured with a couple of environment variables. I was able to configure it using some pig properties, but that's not how the component is typically configured. I'm wondering if there is a way that I can set environment variable for the Job, so that the Pig UDF can pick them up.
Community Manager

Re: tSqoopImport and tSqoopExport do not have "custom" Hadoop Distribution

Hi,
Can you explain us how you configured the tSetEnv component "to no avail"?
To me it should meet your stated expectation.
Elisa