Reading DuckDB TIMESTAMP_NS column returns zero or NULL values via C API in DolphinDB plugin

8 hours ago 2
ARTICLE AD BOX

I am developing a DolphinDB plugin for DuckDB and encountered an issue when reading data of type TIMESTAMP_NS. For example, I have an allTypes table in DuckDB. Executing the following statements:

res_debug = duckdb::query(conn, "SELECT nts::VARCHAR FROM all_types") print(res_debug) res = duckdb::query(conn, "SELECT nts FROM all_types") print(res)

The output is:

CAST(nts AS VARCHAR) ----------------------------- 2023-01-01 12:00:00.000000001 2023-01-01 12:00:01 nts ----------------------------- 1970.01.01T00:00:00.000000000 1970.01.01T00:00:00.000000000

My plugin code for duckdb::query (excerpt) is as follows:

ConstantSP duckdbQuery(Heap* heap, vector<ConstantSP>& args) { // ... parameter validation and connection handling ... // Execute query duckdb_result result; if (duckdb_query(conn, sql.c_str(), &result) == DuckDBError) { string errMsg = duckdb_result_error(&result); duckdb_destroy_result(&result); throw RuntimeException("DuckDB Query Error: " + errMsg); } idx_t rowCount = duckdb_row_count(&result); idx_t colCount = duckdb_column_count(&result); vector<string> colNames; vector<ConstantSP> colVectors; // Pre-create DolphinDB column vectors for (idx_t i = 0; i < colCount; ++i) { colNames.push_back(duckdb_column_name(&result, i)); duckdb_type dType = duckdb_column_type(&result, i); DATA_TYPE ddbType = mapDuckDBToDolphin(dType); colVectors.push_back(Util::createVector(ddbType, rowCount)); } // Fill data for (idx_t col = 0; col < colCount; ++col) { Vector* vec = (Vector*)colVectors[col].get(); duckdb_type type = duckdb_column_type(&result, col); for (idx_t row = 0; row < rowCount; ++row) { if (duckdb_value_is_null(&result, col, row)) { vec->setNull(row); continue; } switch (type) { // ... other cases ... case DUCKDB_TYPE_TIMESTAMP_NS: { int64_t nanos = duckdb_value_int64(&result, col, row); cout << "Debug: Read Nanos from DuckDB: " << nanos << endl; vec->setLong(row, nanos); break; } // ... other cases ... } } } duckdb_destroy_result(&result); return Util::createTable(colNames, colVectors); }

With the above code (using duckdb_value_int64), the debug output shows:

Debug: Read Nanos from DuckDB: 0 Debug: Read Nanos from DuckDB: 0

I then tried using duckdb_value_timestamp_ns:

case DUCKDB_TYPE_TIMESTAMP_NS: { duckdb_timestamp_ns tns = duckdb_value_timestamp_ns(&result, col, row); vec->setLong(row, tns.nanos); break; }

However, the compilation fails because duckdb_value_timestamp_ns is not declared (the DuckDB C API version v1.4.4 I'm using does not have it).

I also attempted to read the value as a VARCHAR:

case DUCKDB_TYPE_TIMESTAMP_NS: { char* strVal = duckdb_value_varchar(&result, col, row); if (strVal != nullptr) { cout << "Debug: Original DuckDB Timestamp String: " << strVal << endl; vec->setString(row, strVal); duckdb_free(strVal); } else { cout << "Debug: DuckDB Timestamp String is NULL" << endl; vec->setNull(row); } break; }

But this prints:

Debug: DuckDB Timestamp String is NULL

It seems that the VARCHAR representation is also NULL, even though the nts::VARCHAR query correctly returns the string representation.

How can I correctly read a TIMESTAMP_NS column from DuckDB using the C API and convert it to DolphinDB's NANOTIMESTAMP type (which expects a 64-bit integer representing nanoseconds since the epoch)? Why does duckdb_value_int64 return 0, and why does duckdb_value_varchar return NULL? Is there a proper way to retrieve nanosecond timestamps with the current DuckDB C API?

Additional details:

DuckDB C API version: 1.4.4

DolphinDB version: 3.0.0.4

OS: Windows 10

Read Entire Article