Profiing a KDB+ database

Highlighted
Employee

Profiing a KDB+ database

Has anyone been able to successfully use the DQ Profiler to profile a KDB+ database (http://kx.com/kdb-plus.php)?

I have been able to configure a tJDBCInput to read data from a table using TOS for DI. However, I believe the commands being issued to retrieve the tables and columns by the DQ Profiler are not compatible with the KDB+ JDBC driver (http://code.kx.com/wiki/Cookbook/InterfacingWithJava#Is_there_a_JDBC_interface.3F)

We can get the source from here: http://www.kx.com/q/c/jdbc.java & http://www.kx.com/q/c/kx/c.java

Lines 522 onwards define the sql interactions.

public class dm implements DatabaseMetaData{private co co;public dm(co x){co=x;}
public ResultSet getCatalogs()throws SQLException{return co.qx("([]TABLE_CAT:enlist `DummyCat)");}
public ResultSet getSchemas()throws SQLException{return co.qx("([]TABLE_SCHEM:enlist `DummySchem)");}
public ResultSet getTableTypes()throws SQLException{return co.qx("([]TABLE_TYPE:`TABLE`VIEW)");}
//public ResultSet getTables(String a,String b,String t,String x[])throws SQLException{return co.qx(
// "raze{([]TABLE_CAT:`DummyCat;TABLE_SCHEM:`DummySchem;TABLE_NAME:system string`a`b x=`VIEW;TABLE_TYPE:x)}each",x);}

Could anyone provide assistance in understanding what the Talend DQ Profiler expects such that the JDBC Driver for KDB+ can be modified accordingly please?

TIA
Employee

Re: Profiing a KDB+ database

Hi Mark


The main DQ code can be found at http://svn.talendforge.org/svn/tos/trunk/org.talend.metadata.managment/src/main/java/org/talend/core...

To retrieve tables, the java.sql.DatabaseMetaData should implement the following methods:

           ResultSet tables = dbJDBCMetadata.getTables(catalogName, schemaPattern, tablePattern, tableType);
String productName = dbJDBCMetadata.getDatabaseProductName();

See the content of the method:
  public List<TdTable> fillTables(Package pack, DatabaseMetaData dbJDBCMetadata, List<String> tableFilter, String tablePattern,

If the returned result set is not empty, we should see the list of tables.

Do you have any error log?
Maybe, try to use the debug level of log4j by editing the log4.xml located in Talend-Studio-r110020-V5.4.0EP/plugins/org.talend.platform.logging_5.4.0.EP_r110020/

Change
	<param name="Threshold" value="info" />

to
	<param name="Threshold" value="debug" />

and
	<param name="Threshold" value="info" />

to
	<param name="Threshold" value="debug" />
Employee

Re: Profiing a KDB+ database

Thanks Seb.

Extract from .log file:

Extract from Log:

Log from "Reload database list":

!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.580
!MESSAGE 2013-10-30 11:21:30,580 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.580
!MESSAGE 2013-10-30 11:21:30,580 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.581
!MESSAGE 2013-10-30 11:21:30,580 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.581
!MESSAGE 2013-10-30 11:21:30,581 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.583
!MESSAGE 2013-10-30 11:21:30,582 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.584
!MESSAGE 2013-10-30 11:21:30,584 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.584
!MESSAGE 2013-10-30 11:21:30,584 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.DatabaseConnectionImpl@6d757e6 (name: KDBplus, visibility: vk_public) (properties: {}, id: _VlencFnXEd-Mbrlij0umsA, comment: , label: KDBplus, synchronised: false, divergency: false) (pathname: null, isCaseSensitive: false, version: null, ContextMode: false, ContextId: null, contextName: null) (DatabaseType: General JDBC, DriverJarPath: C:\q\jdbc.jar, DriverClass: jdbc, URL: jdbc:q:localhost:1234, dbVersionString: null, Port: xxxx, Username: null, Password: , ServerName: , DatasourceName: , FileFieldName: , SID: , SqlSynthax: SQL Syntax, StringQuote: ", NullChar: 000, DbmsId: access_id, ProductId: JDBC, DBRootPath: null, AdditionalParams: , StandardSQL: false, SystemSQL: true, cdcTypeMode: null, SQLMode: <unset>, UiSchema: )


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.585
!MESSAGE 2013-10-30 11:21:30,585 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.AdditionalPropertiesImpl@7c3474b8 (key: CONN_PARA_KEY_HBASE_PROPERTIES, value: [])


!ENTRY org.talend.platform.logging 2 0 2013-10-30 11:21:30.585
!MESSAGE 2013-10-30 11:21:30,585 WARN org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel - Element not updated: org.talend.core.model.metadata.builder.connection.impl.AdditionalPropertiesImpl@7c3474b8 (key: CONN_PARA_KEY_HBASE_PROPERTIES, value: [])


Log from "Reload table list":

!ENTRY org.talend.platform.logging 4 0 2013-10-30 11:22:19.559
!MESSAGE 2013-10-30 11:22:19,558 ERROR org.talend.dq.nodes.DBTableFolderRepNode - java.lang.ArrayIndexOutOfBoundsException: 4

!STACK 0
java.lang.ArrayIndexOutOfBoundsException: 4
at jdbc$rs.getObject(jdbc.java:307)
at jdbc$rs.getString(jdbc.java:315)
at jdbc$rs.getString(jdbc.java:332)
at org.talend.core.model.metadata.DBConnectionFillerImpl.fillTables(DBConnectionFillerImpl.java:1068)
at org.talend.core.model.metadata.MetadataFillFactory.fillTables(MetadataFillFactory.java:192)
at org.talend.core.model.metadata.builder.database.DqRepositoryViewService.loadTables(DqRepositoryViewService.java:366)
at org.talend.core.model.metadata.builder.database.DqRepositoryViewService.getTables(DqRepositoryViewService.java:202)
at org.talend.dq.nodes.DBTableFolderRepNode.createRepositoryNodeTableFolderNode(DBTableFolderRepNode.java:192)
at org.talend.dq.nodes.DBTableFolderRepNode.getChildren(DBTableFolderRepNode.java:138)
at org.talend.dataprofiler.core.ui.views.provider.ResourceViewContentProvider.getChildren(ResourceViewContentProvider.java:170)
at org.eclipse.ui.internal.navigator.extensions.SafeDelegateTreeContentProvider.getChildren(SafeDelegateTreeContentProvider.java:96)
at org.eclipse.ui.internal.navigator.extensions.SafeDelegateTreeContentProvider.getChildren(SafeDelegateTreeContentProvider.java:275)
at org.eclipse.ui.internal.navigator.extensions.SafeDelegateTreeContentProvider.getChildren(SafeDelegateTreeContentProvider.java:94)
at org.eclipse.ui.internal.navigator.NavigatorContentServiceContentProvider$1.run(NavigatorContentServiceContentProvider.java:150)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.ui.internal.navigator.NavigatorContentServiceContentProvider.internalGetChildren(NavigatorContentServiceContentProvider.java:137)
at org.eclipse.ui.internal.navigator.NavigatorContentServiceContentProvider.getChildren(NavigatorContentServiceContentProvider.java:123)
at org.eclipse.jface.viewers.AbstractTreeViewer.getRawChildren(AbstractTreeViewer.java:1348)
at org.eclipse.jface.viewers.TreeViewer.getRawChildren(TreeViewer.java:391)
at org.eclipse.jface.viewers.StructuredViewer.getFilteredChildren(StructuredViewer.java:896)
at org.eclipse.jface.viewers.AbstractTreeViewer.getSortedChildren(AbstractTreeViewer.java:601)
at org.eclipse.jface.viewers.AbstractTreeViewer$1.run(AbstractTreeViewer.java:799)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.jface.viewers.AbstractTreeViewer.createChildren(AbstractTreeViewer.java:778)
at org.eclipse.jface.viewers.TreeViewer.createChildren(TreeViewer.java:644)
at org.eclipse.jface.viewers.AbstractTreeViewer.createChildren(AbstractTreeViewer.java:749)
at org.eclipse.jface.viewers.AbstractTreeViewer.handleTreeExpand(AbstractTreeViewer.java:1444)
at org.eclipse.jface.viewers.TreeViewer.handleTreeExpand(TreeViewer.java:952)
at org.eclipse.jface.viewers.AbstractTreeViewer$4.treeExpanded(AbstractTreeViewer.java:1455)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:132)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1077)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1062)
at org.eclipse.swt.widgets.Tree.wmNotifyChild(Tree.java:7408)
at org.eclipse.swt.widgets.Control.wmNotify(Control.java:5096)
at org.eclipse.swt.widgets.Composite.wmNotify(Composite.java:1887)
at org.eclipse.swt.widgets.Control.WM_NOTIFY(Control.java:4726)
at org.eclipse.swt.widgets.Control.windowProc(Control.java:4215)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:4873)
at org.eclipse.swt.internal.win32.OS.CallWindowProcW(Native Method)
at org.eclipse.swt.internal.win32.OS.CallWindowProc(OS.java:2362)
at org.eclipse.swt.widgets.Tree.callWindowProc(Tree.java:1536)
at org.eclipse.swt.widgets.Tree.WM_LBUTTONDOWN(Tree.java:6303)
at org.eclipse.swt.widgets.Control.windowProc(Control.java:4195)
at org.eclipse.swt.widgets.Tree.windowProc(Tree.java:5844)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:4886)
at org.eclipse.swt.internal.win32.OS.DispatchMessageW(Native Method)
at org.eclipse.swt.internal.win32.OS.DispatchMessage(OS.java:2459)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3655)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2640)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2604)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2438)
at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:671)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:664)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.talend.rcp.intro.Application.start(Application.java:141)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:619)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
at org.eclipse.equinox.launcher.Main.run(Main.java:1407)
at org.eclipse.equinox.launcher.Main.main(Main.java:1383)

!ENTRY org.talend.platform.logging 4 0 2013-10-30 11:22:25.726
!MESSAGE 2013-10-30 11:22:25,725 ERROR org.talend.cwm.compare.ui.actions.ReloadDatabaseAction - org.talend.cwm.compare.exception.ReloadCompareException: java.lang.ArrayIndexOutOfBoundsException: 4

!STACK 0
org.talend.cwm.compare.exception.ReloadCompareException: java.lang.ArrayIndexOutOfBoundsException: 4
at org.talend.cwm.compare.factory.comparisonlevel.CatalogSchemaComparisonLevel.reloadElementOfPackage(CatalogSchemaComparisonLevel.java:313)
at org.talend.cwm.compare.factory.comparisonlevel.CatalogSchemaComparisonLevel.getSavedReloadObject(CatalogSchemaComparisonLevel.java:183)
at org.talend.cwm.compare.factory.comparisonlevel.CatalogSchemaComparisonLevel.compareWithReloadObject(CatalogSchemaComparisonLevel.java:148)
at org.talend.cwm.compare.factory.comparisonlevel.AbstractComparisonLevel.reloadCurrentLevelElement(AbstractComparisonLevel.java:187)
at org.talend.cwm.compare.ui.actions.ReloadDatabaseAction$1$1.run(ReloadDatabaseAction.java:161)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:134)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4041)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3660)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.block(ModalContext.java:173)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:388)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:507)
at org.talend.dataprofiler.core.ui.progress.ProgressUI.popProgressDialog(ProgressUI.java:54)
at org.talend.dataprofiler.core.ui.progress.ProgressUI.popProgressDialog(ProgressUI.java:38)
at org.talend.cwm.compare.ui.actions.ReloadDatabaseAction.run(ReloadDatabaseAction.java:191)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:584)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:501)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:411)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4066)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3657)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2640)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2604)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2438)
at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:671)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:664)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.talend.rcp.intro.Application.start(Application.java:141)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:619)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
at org.eclipse.equinox.launcher.Main.run(Main.java:1407)
at org.eclipse.equinox.launcher.Main.main(Main.java:1383)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
at jdbc$rs.getObject(jdbc.java:307)
at jdbc$rs.getString(jdbc.java:315)
at jdbc$rs.getString(jdbc.java:332)
at org.talend.core.model.metadata.DBConnectionFillerImpl.fillTables(DBConnectionFillerImpl.java:1068)
at org.talend.core.model.metadata.MetadataFillFactory.fillTables(MetadataFillFactory.java:192)
at org.talend.core.model.metadata.builder.database.DqRepositoryViewService.loadTables(DqRepositoryViewService.java:366)
at org.talend.core.model.metadata.builder.database.DqRepositoryViewService.getTables(DqRepositoryViewService.java:202)
at org.talend.cwm.compare.factory.comparisonlevel.CatalogSchemaComparisonLevel.reloadElementOfPackage(CatalogSchemaComparisonLevel.java:279)
... 43 more
One Star

Re: Profiing a KDB+ database

Issue is thrown after trying to access REMARKS in the response from "getTables"

String tableOwner = null;
if (MetadataConnectionUtils.isSybase(dbJDBCMetadata)) {
tableOwner = tables.getString(GetTable.TABLE_SCHEM.name());
}
if (!flag) {
tableComment = tables.getString(GetTable.REMARKS.name()); **************** HERE **********
if (StringUtils.isBlank(tableComment)) {
String selectRemarkOnTable = MetadataConnectionUtils.getCommonQueryStr(productName, tableName);
if (selectRemarkOnTable != null) {
tableComment = executeGetCommentStatement(selectRemarkOnTable, dbJDBCMetadata.getConnection());
}
}
}

re-declaring jdbc.java ~line 500 to:


public ResultSet getTables(String a,String b,String t,String x[])throws SQLException{return co.qx(
"raze{([]TABLE_CAT:enlist `;TABLE_SCHEM:enlist `;REMARKS:enlist `none;TABLE_NAME:system string`a`b x=`VIEW;TABLE_TYPE:x)}each",x);}

will fix it.
Employee

Re: Profiing a KDB+ database

Thanks for sharing Stephen :-)
Employee

Re: Profiing a KDB+ database

Another stumbling block. Once it?s hooked up ? i now need to provide the language logic to query in ?KDB?
The extract below is what is being sent:

(".o.ex";"SELECT COUNT(*) FROM DummyCat.DummySchem.TestTable ")
(".o.ex";"SELECT COUNT(*) FROM DummyCat.DummySchem.TestTable WHERE a IS NULL ")
(".o.ex";"SELECT COUNT(*) FROM (SELECT DISTINCT a FROM DummyCat.DummySchem.TestTable ) A")
(".o.ex";"SELECT COUNT(*) FROM (SELECT a, COUNT(*) FROM DummyCat.DummySchem.TestTable GROUP BY a HAVING COUNT(*) = 1) AS myquery")
(".o.ex";"SELECT COUNT(*) FROM (SELECT a, COUNT(*) FROM DummyCat.DummySchem.TestTable m GROUP BY a HAVING COUNT(*) > 1) AS myquery")

And I?ll need to convert that into something KDB can handle. KDB doesn?t support ?having? as such...

Maybe i can do that with a mapping file???
Employee

Re: Profiing a KDB+ database

Hi Mark,

most SQL templates can be modified in the indicators:
https://help.talend.com/search/all?query=Managing+system+indicators

But I admit that a few of them may need some java code from our side.
In this case, I encourage the users to use a User Defined Indicator instead of the system until we can fix the java code.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Introduction to Talend Open Studio for Data Quality

Find out about Talend Open Studio for Data Quality

Watch Now

Enabling Data Governance

Learn how to enable Data Governance

Watch Now

The Definitive Guide to Government Data Quality

Take a peek at the definitive guide to Government Data Quality

Read