Starrocks configurations

Model Configuration

A dbt model can be configured using the following syntax:

Project file

dbt_project.yml

models:
  <resource-path>:
    materialized: table       // table or view or materialized_view
    keys: ['id', 'name', 'some_date']
    table_type: 'PRIMARY'     // PRIMARY or DUPLICATE or UNIQUE
    distributed_by: ['id']
    buckets: 3                // default 10
    partition_by: ['some_date']
    partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"]
    properties: [{"replication_num":"1", "in_memory": "true"}]
    refresh_method: 'async' // only for materialized view default manual

Property file

models/properties.yml

models:
  - name: <model-name>
    config:
      materialized: table       // table or view or materialized_view
      keys: ['id', 'name', 'some_date']
      table_type: 'PRIMARY'     // PRIMARY or DUPLICATE or UNIQUE
      distributed_by: ['id']
      buckets: 3                // default 10
      partition_by: ['some_date']
      partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"]
      properties: [{"replication_num":"1", "in_memory": "true"}]
      refresh_method: 'async' // only for materialized view default manual

Config block

models/<model_name>.sql

{{ config(
    materialized = 'table',
    keys=['id', 'name', 'some_date'],
    table_type='PRIMARY',
    distributed_by=['id'],
    buckets=3,
    partition_by=['some_date'],
    ....
) }}

Configuration Description

Option	Description
materialized	How the model will be materialized into Starrocks. Supports view, table, incremental, ephemeral, and materialized_view.
keys	Which columns serve as keys.
table_type	Table type, supported are PRIMARY or DUPLICATE or UNIQUE.
distributed_by	Specifies the column of data distribution. If not specified, it defaults to random.
buckets	The bucket number in one partition. If not specified, it will be automatically inferred.
partition_by	The partition column list.
partition_by_init	The partition rule or some real partitions item.
properties	The table properties configuration of Starrocks. (Starrocks table properties)
refresh_method	How to refresh materialized views.

Read From Catalog

First you need to add this catalog to starrocks. The following is an example of hive.

CREATE EXTERNAL CATALOG `hive_catalog`
PROPERTIES (
    "hive.metastore.uris"  =  "thrift://127.0.0.1:8087",
    "type"="hive"
);

How to add other types of catalogs can be found in the documentation. Catalog Overview Then write the sources.yaml file.

sources:
  - name: external_example
    schema: hive_catalog.hive_db
    tables:
      - name: hive_table_name

Finally, you might use below marco quote

{{ source('external_example', 'hive_table_name') }}

Was this page helpful?

Privacy policyCreate a GitHub issue

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.