MISP parser: added functionality to honor/filter the "to_ids" attribute of MISP

[DK1MI]

Added the following to honor the "to_ids" attribute of MISP:

• added the new field "misp.to_ids" to harmonization.conf (Boolean)
• added the new parameter "only_ids" to the MISP parser
• MISP parser now inserts the MISP attribute "to_ids" into the resulting IntelMQ events
• When parameter "only_ids" is set to "true", the MISP parser only processes and forwards MISP events if their "to_ids" attribute's value is true

[ghost]

This introduces a new field in the harmonization (internal data format) -> @aaronkaplan

[Rafiot]

It seems fine to me.

[ghost]

It seems fine to me.

Thanks!

[ghost]

@exitnode Can you please better explain what the to_ids field is for (the given explanation in harmonization.conf is way too generic`) and why we need a separate field for it in IntelMQ?

[DK1MI]

@exitnode Can you please better explain what the to_ids field is for (the given explanation in harmonization.conf is way too generic`) and why we need a separate field for it in IntelMQ?

This is from the MISP core format documentation:

to_ids represents whether the attribute is meant to be actionable.
Actionable defined attributes that can be used in automated processes
as a pattern for detection in Local or Network Intrusion Detection
System, log analysis tools or even filtering mechanisms

We use the "to_ids" flag inside of MISP to decide if an IOC will be later on be included in a blocking list e.g. for a proxy server. We use this to manually verify what will be blocked and what not while working on the MISP events.

Now we can filter inside IntelMQ if an event' property is meant for blocking or not and therefore put it in a pipeline that eventually generates the blocking list.

FYI: I'm working on additional enhancements to the MISP collector that will also include this property and some more functionality. As soon as it is stable enough, I will send a PR for it, too.

[ghost]

If I got the meaning of the to_ids field right, I think this is similar to the discussion #758. Both fields have equivalent meanings, i.e. that the IoC is actionable. IMHO we should only have one field, not one field per input/output as long as they do not conflict (I don't see this risk at the moment)

[ghost]

For me, this is not descriptive enough. I can't understand the meaning of the field from this text.