EF persistence for failed messages and streamline known endpoints - #5638
Conversation
8f7ce88 to
9346fb1
Compare
| const string CatalogName = "ServiceControlFullTextCatalog"; | ||
|
|
||
| // Both statements are guarded: an instance without Full-Text Search installed still migrates, | ||
| // it just has no full text index. They also cannot run inside a transaction, so the migration |
There was a problem hiding this comment.
do we actually need to support instances without FTS?
If so we probably need to surface that as an error and have a way to recreate this index so that there is an upgrade path.
There was a problem hiding this comment.
I am not sure, I remember as part of the spikes we allowed customers to configure it.
@warwickschroeder any thoughts?
There was a problem hiding this comment.
We can make a decision, later on this?
Without FTS, they would get no search at all.
I don't think we want to support LIKE %blah% !
From our spikes FTS was not an issue at all, so i think we can make it a required thing.
There was a problem hiding this comment.
Not having it as an option gives the user the option to not having to store the body in the DB and only use cheap external storage. That may be worth it?
This change introduces the core EF Core schema and entities for storing failed messages, including their associated groups and retry information. It also sets up full-text search capabilities for failed message headers, bodies, and message types in both PostgreSQL and SQL Server. The separate 'insert-only' table and background reconciler for KnownEndpoints have been removed, simplifying how known endpoints are managed. A new `MaxBodySizeToStore` setting is added to control message body storage.
This change completes the error ingestion process for PostgreSQL and SQL Server persisters, enabling robust and concurrent storage of failed messages. It introduces a detailed design document (`error-ingestion-design.md`) outlining the complex write path.
Adds a `RetentionSweeper` background service to automatically delete resolved and archived failed messages that have exceeded the configured retention period. The sweeper runs periodically in batches, handling external message bodies and tolerating body deletion failures.
9346fb1 to
ed0b1a6
Compare
warwickschroeder
left a comment
warwickschroeder
left a comment
There was a problem hiding this comment.
Review so far. Not finished yet. Will review the rest tonight/tomorrow.
| <Compile Include="..\ServiceControl.Persistence.Tests\**\*.cs" LinkBase="Shared" /> | ||
|
|
||
| <!-- Tests of behaviour specific to the EF Core persisters, shared by their two test projects only --> | ||
| <Compile Remove="..\ServiceControl.Persistence.Tests\EFCore\**\*.cs" /> |
There was a problem hiding this comment.
- I'm guessing this is why the change in Limit the wildcard remove check to items that reference source packages repo-integrity-action#153 was made, which I don't disagree with, but does this indicate that the tests involved should be further split into different sources so a Compile Include+Remove pattern is not needed?
There was a problem hiding this comment.
@DavidBoike the reason to go with this pattern is so we can share tests between both PostgreSQL and SqlServer tests.
What do you mean by further split?
4 participants
This change introduces the core EF Core schema and entities for storing failed messages, including their associated groups and retry information. It also sets up full-text search capabilities for failed message headers, bodies, and message types in both PostgreSQL and SQL Server.
The separate 'insert-only' table and background reconciler for KnownEndpoints have been removed, simplifying how known endpoints are managed. A new
MaxBodySizeToStoresetting is added to control message body storage.