tsqoopImport issue with postgresql

One Star

tsqoopImport issue with postgresql

Hello,
i want to import a table (country) from postgresql to hdfs using tSqoopImport componant.
I had a error with my tsqoopImport "Ident authentication failed for user "postgres" :
Somebody had a idea ?
Thks for your help.
-----------------------------------------------------------------------------------------------
Starting job testsqoop at 16:08 18/03/2014.
connecting to socket on port 4016
connected
.--+-------+-----------.
| tLogRow_1 |
|=-+-------+----------=|
|id|isocode|countryname|
|=-+-------+----------=|
|1 |LU |Luxembourg |
|2 |FR |France |
|3 |BE |Belgique |
'--+-------+-----------'
: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
: org.apache.sqoop.ConnFactory - Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
: org.apache.sqoop.ConnFactory - Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
: org.apache.sqoop.manager.DefaultManagerFactory - Trying with scheme: jdbcSmiley Tongueostgresql:
: org.apache.sqoop.manager.SqlManager - Using default fetchSize of 1000
: org.apache.sqoop.ConnFactory - Instantiated ConnManager org.apache.sqoop.manager.PostgresqlManager@2e4ca60c
: org.apache.sqoop.tool.CodeGenTool - Beginning code generation
: org.apache.sqoop.manager.SqlManager - No connection paramenters specified. Using regular API for making connection.
: org.apache.sqoop.manager.SqlManager - Using fetchSize for next query: 1000
: org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM "country" AS t LIMIT 1
: org.apache.sqoop.orm.ClassWriter - selected columns:
: org.apache.sqoop.orm.ClassWriter - id
: org.apache.sqoop.orm.ClassWriter - isocode
: org.apache.sqoop.orm.ClassWriter - countryname
: org.apache.sqoop.orm.ClassWriter - Writing source file: \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.java
: org.apache.sqoop.orm.ClassWriter - Table name: country
: org.apache.sqoop.orm.ClassWriter - Columns: id:-5, isocode:12, countryname:12,
: org.apache.sqoop.orm.ClassWriter - sourceFilename is country.java
: org.apache.sqoop.orm.CompilationManager - Found existing \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\
: org.apache.sqoop.orm.CompilationManager - $HADOOP_HOME is not set
: org.apache.sqoop.orm.CompilationManager - Adding source file: \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.java
: org.apache.sqoop.orm.CompilationManager - Invoking javac with args:
: org.apache.sqoop.orm.CompilationManager - -sourcepath
: org.apache.sqoop.orm.CompilationManager - \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\
: org.apache.sqoop.orm.CompilationManager - -d
: org.apache.sqoop.orm.CompilationManager - \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\
: org.apache.sqoop.orm.CompilationManager - -classpath
: org.apache.sqoop.orm.CompilationManager - C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/commons-cli-1.2.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/commons-configuration-1.6.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/commons-io-1.4.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/commons-lang-2.4.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/commons-logging-1.1.1.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/dom4j-1.6.1.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/hadoop-core-1.2.0.1.3.0.0-107.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/jackson-core-asl-1.7.3.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/jackson-mapper-asl-1.7.3.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/jersey-core-1.8.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/jetty-util-6.1.26.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/log4j-1.2.17.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/postgresql-9.2-1002.jdbc4.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/postgresql-9.2-1003.jdbc3.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/sqoop-1.4.2.21.jar;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/sqoop-1.4.3.1.3.0.0-107.jar;.;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/classes;C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib;;/C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/hadoop-core-1.2.0.1.3.0.0-107.jar;/C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/sqoop-1.4.2.21.jar
Note: \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
: org.apache.sqoop.orm.CompilationManager - Could not rename \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.java to C:\huy\TOS_BD-r111943-V5.4.1\.\country.java
java.io.IOException: Destination 'C:\huy\TOS_BD-r111943-V5.4.1\.\country.java' already exists
at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:1811)
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:227)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:83)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at fer.testsqoop_0_1.testsqoop.tSqoopImport_1Process(testsqoop.java:1597)
at fer.testsqoop_0_1.testsqoop.tPostgresqlConnection_1Process(testsqoop.java:871)
at fer.testsqoop_0_1.testsqoop.tLibraryLoad_2Process(testsqoop.java:719)
at fer.testsqoop_0_1.testsqoop.tLibraryLoad_1Process(testsqoop.java:609)
at fer.testsqoop_0_1.testsqoop.tHDFSDelete_1Process(testsqoop.java:499)
at fer.testsqoop_0_1.testsqoop.runJobInTOS(testsqoop.java:1859)
at fer.testsqoop_0_1.testsqoop.main(testsqoop.java:1724)
: org.apache.sqoop.orm.CompilationManager - Writing jar file: \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.jar
: org.apache.sqoop.orm.CompilationManager - Scanning for .class files in directory: \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae
: org.apache.sqoop.orm.CompilationManager - Got classfile: \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.class -> country.class
: org.apache.sqoop.orm.CompilationManager - Finished writing jar file \tmp\sqoop-rui.martins\compile\c2546c71efafaec40852209062fc83ae\country.jar
: org.apache.sqoop.manager.PostgresqlManager - It looks like you are importing from postgresql.
: org.apache.sqoop.manager.PostgresqlManager - This transfer can be faster! Use the --direct
: org.apache.sqoop.manager.PostgresqlManager - option to exercise a postgresql-specific fast path.
: org.apache.sqoop.manager.CatalogQueryManager - Retrieving primary key for table 'country' with query SELECT col.ATTNAME FROM PG_CATALOG.PG_NAMESPACE sch, PG_CATALOG.PG_CLASS tab, PG_CATALOG.PG_ATTRIBUTE col, PG_CATALOG.PG_INDEX ind WHERE sch.OID = tab.RELNAMESPACE AND tab.OID = col.ATTRELID AND tab.OID = ind.INDRELID AND sch.NSPNAME = (SELECT CURRENT_SCHEMA()) AND tab.RELNAME = 'country' AND col.ATTNUM = ANY(ind.INDKEY) AND ind.INDISPRIMARY
: org.apache.sqoop.manager.CatalogQueryManager - Retrieving primary key for table 'country' with query SELECT col.ATTNAME FROM PG_CATALOG.PG_NAMESPACE sch, PG_CATALOG.PG_CLASS tab, PG_CATALOG.PG_ATTRIBUTE col, PG_CATALOG.PG_INDEX ind WHERE sch.OID = tab.RELNAMESPACE AND tab.OID = col.ATTRELID AND tab.OID = ind.INDRELID AND sch.NSPNAME = (SELECT CURRENT_SCHEMA()) AND tab.RELNAME = 'country' AND col.ATTNUM = ANY(ind.INDKEY) AND ind.INDISPRIMARY
: org.apache.sqoop.mapreduce.ImportJobBase - Beginning import of country
: org.apache.sqoop.mapreduce.DataDrivenImportJob - Using table class: country
: org.apache.sqoop.mapreduce.DataDrivenImportJob - Using InputFormat: class com.cloudera.sqoop.mapreduce.db.DataDrivenDBInputFormat
: org.apache.sqoop.mapreduce.JobBase - Adding to job classpath: file:/C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/sqoop-1.4.2.21.jar
: org.apache.sqoop.mapreduce.JobBase - Adding to job classpath: file:/C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/postgresql-9.2-1002.jdbc4.jar
: org.apache.sqoop.mapreduce.JobBase - Adding to job classpath: file:/C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/sqoop-1.4.2.21.jar
: org.apache.sqoop.mapreduce.JobBase - Adding to job classpath: file:/C:/huy/TOS_BD-r111943-V5.4.1/workspace/FER/.Java/lib/sqoop-1.4.2.21.jar
: org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies.
: org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
: org.apache.hadoop.mapred.JobClient - Running job: job_201403180706_0004
: org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
: org.apache.hadoop.mapred.JobClient - Task Id : attempt_201403180706_0004_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:193)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
... 9 more
Caused by: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:293)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:184)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
... 10 more
: org.apache.hadoop.mapred.JobClient - Task Id : attempt_201403180706_0004_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:193)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
... 9 more
Caused by: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:293)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:184)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
... 10 more
: org.apache.hadoop.mapred.JobClient - Task Id : attempt_201403180706_0004_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:193)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
... 9 more
Caused by: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:293)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:184)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
... 10 more
: org.apache.hadoop.mapred.JobClient - Job complete: job_201403180706_0004
: org.apache.hadoop.mapred.JobClient - Counters: 6
: org.apache.hadoop.mapred.JobClient - Job Counters
: org.apache.hadoop.mapred.JobClient - SLOTS_MILLIS_MAPS=47041
: org.apache.hadoop.mapred.JobClient - Total time spent by all reduces waiting after reserving slots (ms)=0
: org.apache.hadoop.mapred.JobClient - Total time spent by all maps waiting after reserving slots (ms)=0
: org.apache.hadoop.mapred.JobClient - Launched map tasks=4
: org.apache.hadoop.mapred.JobClient - SLOTS_MILLIS_REDUCES=0
: org.apache.hadoop.mapred.JobClient - Failed map tasks=1
: org.apache.sqoop.mapreduce.ImportJobBase - Transferred 0 bytes in 59,7574 seconds (0 bytes/sec)
: org.apache.sqoop.mapreduce.ImportJobBase - Retrieved 0 records.
: org.apache.sqoop.tool.ImportTool - Error during import: Import job failed!
Exception in component tSqoopImport_1
java.lang.Exception: The Sqoop import job has failed
at fer.testsqoop_0_1.testsqoop.tSqoopImport_1Process(testsqoop.java:1599)
at fer.testsqoop_0_1.testsqoop.tPostgresqlConnection_1Process(testsqoop.java:871)
at fer.testsqoop_0_1.testsqoop.tLibraryLoad_2Process(testsqoop.java:719)
at fer.testsqoop_0_1.testsqoop.tLibraryLoad_1Process(testsqoop.java:609)
at fer.testsqoop_0_1.testsqoop.tHDFSDelete_1Process(testsqoop.java:499)
at fer.testsqoop_0_1.testsqoop.runJobInTOS(testsqoop.java:1859)
at fer.testsqoop_0_1.testsqoop.main(testsqoop.java:1724)
disconnected
Job testsqoop ended at 16:10 18/03/2014.
Highlighted
Employee

Re: tsqoopImport issue with postgresql

Hi hvuong?
From error log,seems to be that you have no authentication.
If you talend product install not the same as postgreSQL DB Server,you must open remote connect in postgresSQL.
jjzhou
One Star

Re: tsqoopImport issue with postgresql

hi jjzhou,
thks for your response.
But i m don't understand because in my job i had a tPostGreSqlConnection and it is ok, i can connect with my database.
I don't understand why my tSqoopImport doesnt work ?
See my screenshot below.
Thks.