Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions benchmarks/Eftdb.Benchmarks/Eftdb.Benchmarks.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@

<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.15.8" />
<PackageReference Include="Testcontainers.PostgreSql" Version="4.9.0" />
<PackageReference Include="Z.EntityFramework.Extensions.EFCore" Version="10.105.0" />
<PackageReference Include="Testcontainers.PostgreSql" Version="4.11.0" />
<PackageReference Include="Z.EntityFramework.Extensions.EFCore" Version="10.105.3" />
</ItemGroup>

<ItemGroup>
Expand Down
3 changes: 1 addition & 2 deletions benchmarks/Eftdb.Benchmarks/WriteRecordsBenchmarkBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@ public abstract class WriteRecordsBenchmarkBase<T> where T : class
public int MaxBatchSize;
public int NumberOfWorkers;

private readonly PostgreSqlContainer _dbContainer = new PostgreSqlBuilder()
.WithImage("timescale/timescaledb:latest-pg17")
private readonly PostgreSqlContainer _dbContainer = new PostgreSqlBuilder("timescale/timescaledb:latest-pg17")
.WithDatabase("benchmark_tests_db")
.WithUsername("test_user")
.WithPassword("test_password")
Expand Down
94 changes: 94 additions & 0 deletions docs/data-annotations/retention-policies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Retention Policies

A retention policy automatically drops old chunks from a hypertable or continuous aggregate on a scheduled basis. This keeps storage consumption bounded without requiring manual intervention and is the standard approach for managing time-series data lifecycle in TimescaleDB.

Each hypertable or continuous aggregate supports at most one retention policy.

[See also: add_retention_policy](https://docs.tigerdata.com/api/latest/data_retention/add_retention_policy/)

## Drop Modes

Two mutually exclusive drop modes are available:

- **`DropAfter`**: Drops chunks whose data falls outside a time window relative to the current time. This is the standard mode.
- **`DropCreatedBefore`**: Drops chunks created before a specified interval ago, regardless of the data they contain.

Exactly one of `DropAfter` or `DropCreatedBefore` must be specified. Providing both or neither raises an exception.

## Basic Example

Here is a complete example of configuring a retention policy on an `ApplicationLog` hypertable using `DropAfter`.

```csharp
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.Hypertable;
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.RetentionPolicy;
using Microsoft.EntityFrameworkCore;

[Hypertable(nameof(Time), ChunkTimeInterval = "1 day")]
[PrimaryKey(nameof(Id), nameof(Time))]
[RetentionPolicy("30 days")]
public class ApplicationLog
{
public Guid Id { get; set; }
public DateTime Time { get; set; }
public string ServiceName { get; set; } = string.Empty;
public string Level { get; set; } = string.Empty;
public string Message { get; set; } = string.Empty;
}
```

## Using `DropCreatedBefore`

Pass `null` as the first argument and provide `dropCreatedBefore` as a named argument:

```csharp
[Hypertable(nameof(Time), ChunkTimeInterval = "1 day")]
[PrimaryKey(nameof(Id), nameof(Time))]
[RetentionPolicy(dropCreatedBefore: "30 days")]
public class ApiRequestLog
{
public Guid Id { get; set; }
public DateTime Time { get; set; }
public string Path { get; set; } = string.Empty;
public int StatusCode { get; set; }
}
```

> :warning: **Note:** Due to a known bug in TimescaleDB ([#9446](https://github.com/timescale/timescaledb/issues/9446)), `alter_job` fails when used with `DropCreatedBefore` policies. The library works around this by skipping the `alter_job` call for `DropCreatedBefore` policies. As a result, job scheduling parameters (`ScheduleInterval`, `MaxRuntime`, `MaxRetries`, `RetryPeriod`) are accepted by the API but have no effect at the database level when `DropCreatedBefore` is used.

## Complete Example

```csharp
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.Hypertable;
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.RetentionPolicy;
using Microsoft.EntityFrameworkCore;

[Hypertable(nameof(Time), ChunkTimeInterval = "1 day")]
[PrimaryKey(nameof(Id), nameof(Time))]
[RetentionPolicy("30 days",
InitialStart = "2025-10-01T03:00:00Z",
ScheduleInterval = "1 day",
MaxRuntime = "30 minutes",
MaxRetries = 3,
RetryPeriod = "5 minutes")]
public class ApplicationLog
{
public Guid Id { get; set; }
public DateTime Time { get; set; }
public string ServiceName { get; set; } = string.Empty;
public string Level { get; set; } = string.Empty;
public string Message { get; set; } = string.Empty;
}
```

## Supported Parameters

| Parameter | Description | Type | Database Type | Default Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------- | ------------- | ------------------------------------------- |
| `DropAfter` | The interval after which chunks are dropped. Mutually exclusive with `DropCreatedBefore`. Can be passed as the first positional argument. | `string?` | `INTERVAL` | — |
| `DropCreatedBefore` | The interval before which chunks created are dropped. Based on chunk creation time. Only supports `INTERVAL`. Not available for integer-based time columns. Mutually exclusive with `DropAfter`. | `string?` | `INTERVAL` | — |
| `InitialStart` | The first time the policy job is scheduled to run, specified as a UTC date-time string in ISO 8601 format. If `null`, the first run is based on the `ScheduleInterval`. | `string?` | `TIMESTAMPTZ` | `null` |
| `ScheduleInterval` | The interval at which the retention policy job runs. | `string?` | `INTERVAL` | `'1 day'` |
| `MaxRuntime` | The maximum amount of time the job is allowed to run before being stopped. If `null`, there is no time limit. | `string?` | `INTERVAL` | `'00:00:00'` |
| `MaxRetries` | The number of times the job is retried if it fails. | `int` | `INTEGER` | `-1` |
| `RetryPeriod` | The amount of time the scheduler waits between retries of a failed job. | `string?` | `INTERVAL` | Equal to the `scheduleInterval` by default. |
110 changes: 110 additions & 0 deletions docs/fluent-api/retention-policies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Retention Policies

A retention policy automatically drops old chunks from a hypertable or continuous aggregate on a scheduled basis. This keeps storage consumption bounded without requiring manual intervention and is the standard approach for managing time-series data lifecycle in TimescaleDB.

Each hypertable or continuous aggregate supports at most one retention policy.

[See also: add_retention_policy](https://docs.tigerdata.com/api/latest/data_retention/add_retention_policy/)

## Drop Modes

Two mutually exclusive drop modes are available:

- **`dropAfter`**: Drops chunks whose data falls outside a time window relative to the current time. This is the standard mode.
- **`dropCreatedBefore`**: Drops chunks created before a specified interval ago, regardless of the data they contain.

Exactly one of `dropAfter` or `dropCreatedBefore` must be specified. Providing both or neither raises an exception.

## Basic Example

```csharp
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.RetentionPolicy;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;

public class ApplicationLogConfiguration : IEntityTypeConfiguration<ApplicationLog>
{
public void Configure(EntityTypeBuilder<ApplicationLog> builder)
{
builder.HasKey(x => new { x.Id, x.Time });

builder.IsHypertable(x => x.Time)
.WithChunkTimeInterval("1 day");

// Drop chunks older than 30 days, running the job daily
builder.WithRetentionPolicy(
dropAfter: "30 days",
scheduleInterval: "1 day");
}
}
```

## Using `dropCreatedBefore`

```csharp
public class ApiRequestLogConfiguration : IEntityTypeConfiguration<ApiRequestLog>
{
public void Configure(EntityTypeBuilder<ApiRequestLog> builder)
{
builder.HasKey(x => new { x.Id, x.Time });

builder.IsHypertable(x => x.Time)
.WithChunkTimeInterval("1 day");

// Drop chunks created more than 30 days ago
builder.WithRetentionPolicy(
dropCreatedBefore: "30 days",
scheduleInterval: "1 day");
}
}
```

> :warning: **Note:** Due to a known bug in TimescaleDB ([#9446](https://github.com/timescale/timescaledb/issues/9446)), `alter_job` fails when used with `drop_created_before` policies. The library works around this by skipping the `alter_job` call for `drop_created_before` policies. As a result, job scheduling parameters (`scheduleInterval`, `maxRuntime`, `maxRetries`, `retryPeriod`) are accepted by the API but have no effect at the database level when `dropCreatedBefore` is used.

## Complete Example

```csharp
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.RetentionPolicy;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;

public class ApplicationLogConfiguration : IEntityTypeConfiguration<ApplicationLog>
{
public void Configure(EntityTypeBuilder<ApplicationLog> builder)
{
builder.HasKey(x => new { x.Id, x.Time });

builder.IsHypertable(x => x.Time)
.WithChunkTimeInterval("1 day");

builder.WithRetentionPolicy(
dropAfter: "30 days",
initialStart: new DateTime(2025, 10, 1, 3, 0, 0, DateTimeKind.Utc),
scheduleInterval: "1 day",
maxRuntime: "30 minutes",
maxRetries: 3,
retryPeriod: "5 minutes");
}
}

public class ApplicationLog
{
public Guid Id { get; set; }
public DateTime Time { get; set; }
public string ServiceName { get; set; } = string.Empty;
public string Level { get; set; } = string.Empty;
public string Message { get; set; } = string.Empty;
}
```

## Supported Parameters

| Parameter | Description | Type | Database Type | Default Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------- | ------------- | ------------------------------------------- |
| `dropAfter` | The interval after which chunks are dropped. Mutually exclusive with `dropCreatedBefore`. | `string?` | `INTERVAL` | — |
| `dropCreatedBefore` | The interval before which chunks created are dropped. Based on chunk creation time. Only supports `INTERVAL`. Not available for integer-based time columns. Mutually exclusive with `dropAfter`. | `string?` | `INTERVAL` | — |
| `initialStart` | The first time the policy job is scheduled to run, as a UTC `DateTime`. If `null`, the first run is based on the `scheduleInterval`. | `DateTime?` | `TIMESTAMPTZ` | `null` |
| `scheduleInterval` | The interval at which the retention policy job runs. | `string?` | `INTERVAL` | `'1 day'` |
| `maxRuntime` | The maximum amount of time the job is allowed to run before being stopped. If `null`, there is no time limit. | `string?` | `INTERVAL` | `'00:00:00'` |
| `maxRetries` | The number of times the job is retried if it fails. | `int?` | `INTEGER` | `-1` |
| `retryPeriod` | The amount of time the scheduler waits between retries of a failed job. | `string?` | `INTERVAL` | Equal to the `scheduleInterval` by default. |
42 changes: 21 additions & 21 deletions samples/Eftdb.Samples.CodeFirst/Eftdb.Samples.CodeFirst.csproj
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<AssemblyName>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.CodeFirst</AssemblyName>
<RootNamespace>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.CodeFirst</RootNamespace>
</PropertyGroup>
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<AssemblyName>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.CodeFirst</AssemblyName>
<RootNamespace>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.CodeFirst</RootNamespace>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="EFCore.NamingConventions" Version="10.0.0-rc.2" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="10.0.0" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="EFCore.NamingConventions" Version="10.0.1" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="10.0.5" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Eftdb.Samples.Shared\Eftdb.Samples.Shared.csproj" />
<ProjectReference Include="..\..\src\Eftdb.Design\Eftdb.Design.csproj" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Eftdb.Samples.Shared\Eftdb.Samples.Shared.csproj" />
<ProjectReference Include="..\..\src\Eftdb.Design\Eftdb.Design.csproj" />
</ItemGroup>

<ItemGroup>
<Content Include="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
<ItemGroup>
<Content Include="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>

</Project>
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk">

<ItemGroup>
<ProjectReference Include="..\..\src\Eftdb.Design\Eftdb.Design.csproj" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\Eftdb.Design\Eftdb.Design.csproj" />
</ItemGroup>

<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<AssemblyName>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.DatabaseFirst</AssemblyName>
<RootNamespace>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.DatabaseFirst</RootNamespace>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<AssemblyName>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.DatabaseFirst</AssemblyName>
<RootNamespace>CmdScale.EntityFrameworkCore.TimescaleDB.Samples.DatabaseFirst</RootNamespace>
</PropertyGroup>

</Project>
5 changes: 5 additions & 0 deletions samples/Eftdb.Samples.DatabaseFirst/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

This project demonstrates how to use the **Database-First** approach with [TimescaleDB](https://www.timescale.com/) using the `CmdScale.EntityFrameworkCore.TimescaleDB` package.


> [!WARNING]
> Currently the `dotnet ef dbcontext scaffold` command can't be tested because of an issue in `efcore` (see https://github.com/dotnet/efcore/issues/37201).
> <br />TODO: Test as soon as there is a fix available

---

## Required NuGet Packages
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
using CmdScale.EntityFrameworkCore.TimescaleDB.Abstractions;
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.ContinuousAggregate;
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.ContinuousAggregatePolicy;
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.RetentionPolicy;
using CmdScale.EntityFrameworkCore.TimescaleDB.Samples.Shared.Models;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;

namespace CmdScale.EntityFrameworkCore.TimescaleDB.Samples.Shared.Configurations
{
public class ApiRequestAggregateConfiguration : IEntityTypeConfiguration<ApiRequestAggregate>
{
public void Configure(EntityTypeBuilder<ApiRequestAggregate> builder)
{
builder.HasNoKey();
builder.IsContinuousAggregate<ApiRequestAggregate, ApiRequestLog>("api_request_hourly_stats", "1 hour", x => x.Time, true)
.AddAggregateFunction(x => x.AverageDurationMs, x => x.DurationMs, EAggregateFunction.Avg)
.AddAggregateFunction(x => x.MaxDurationMs, x => x.DurationMs, EAggregateFunction.Max)
.AddAggregateFunction(x => x.MinDurationMs, x => x.DurationMs, EAggregateFunction.Min)
.AddGroupByColumn(x => x.ServiceName)
.WithRefreshPolicy(startOffset: "2 days", endOffset: "1 hour", scheduleInterval: "1 hour");
builder.WithRetentionPolicy(
dropAfter: "90 days",
scheduleInterval: "1 day",
maxRetries: 3,
retryPeriod: "15 minutes");
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.Hypertable;
using CmdScale.EntityFrameworkCore.TimescaleDB.Configuration.RetentionPolicy;
using CmdScale.EntityFrameworkCore.TimescaleDB.Samples.Shared.Models;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;

namespace CmdScale.EntityFrameworkCore.TimescaleDB.Samples.Shared.Configurations
{
public class ApiRequestLogConfiguration : IEntityTypeConfiguration<ApiRequestLog>
{
public void Configure(EntityTypeBuilder<ApiRequestLog> builder)
{
builder.ToTable("ApiRequestLogs");
builder.HasNoKey()
.IsHypertable(x => x.Time)
.WithChunkTimeInterval("1 day");
builder.WithRetentionPolicy(
dropCreatedBefore: "30 days",
scheduleInterval: "1 day",
maxRetries: 5,
retryPeriod: "10 minutes");
}
}
}
11 changes: 11 additions & 0 deletions samples/Eftdb.Samples.Shared/Models/ApiRequestAggregate.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
namespace CmdScale.EntityFrameworkCore.TimescaleDB.Samples.Shared.Models
{
public class ApiRequestAggregate
{
public DateTime TimeBucket { get; set; }
public string ServiceName { get; set; } = string.Empty;
public double AverageDurationMs { get; set; }
public double MaxDurationMs { get; set; }
public double MinDurationMs { get; set; }
}
}
Loading
Loading