Materialize v0.87

v0.87

Sources and sinks

• Add support for handling batched events formatted as NDJSON in the webhook source.

  CREATE SOURCE webhook_json IN CLUSTER quickstart FROM WEBHOOK
  BODY FORMAT JSON;
  
  -- Send multiple events delimited by newlines to the webhook source.
  HTTP POST to 'webhook_json'
    { 'event_type': 'foo' }
    { 'event_type': 'bar' }
  
  SELECT COUNT(*) FROM webhook_json;
  2
  

• Allow specifying a default AWS PrivateLink connection when creating a Kafka connection over PrivateLink using the AWS PRIVATELINK top-level option. The default connection will be used to connect to all brokers, and is exclusive with the BROKER and BROKERS options.

  CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
      SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0e123abc123198abc',
      AVAILABILITY ZONES ('use1-az1')
  );
  
  CREATE CONNECTION kafka_connection TO KAFKA (
      AWS PRIVATELINK (PORT 30292)
      SECURITY PROTOCOL = 'SASL_PLAINTEXT',
      SASL MECHANISMS = 'SCRAM-SHA-256',
      SASL USERNAME = 'foo',
      SASL PASSWORD = SECRET red_panda_password
  );
  

• Add topic to the mz_internal.mz_kafka_sources system catalog table. This column contains the name of the Kafka topic the source is reading from.

SQL

• Support user-configured data retention for tables via the RETAIN HISTORY syntax.

Bug fixes and other improvements

• Add a node_ids output modifier for EXPLAIN PHYSICAL PLAN statements, to show the unique ID of each subplan in the plan (#24944).