Skip to content

FIELD_NAMING_NORMALIZATION.md

Latest commit

 

History

History
134 lines (89 loc) · 7.76 KB

FIELD_NAMING_NORMALIZATION.md

File metadata and controls

134 lines (89 loc) · 7.76 KB
Raw

Field Naming Normalization Guide

Raw FortiOS log fields — srcip, dstip, proto, action, devid, and hundreds more — are meaningful within Fortinet's own ecosystem, but become opaque the moment logs land in a multi-source SIEM alongside firewall, endpoint, identity, and cloud telemetry. Field name normalization is the process of mapping vendor-specific field names to a shared schema, enabling cross-source correlation, unified detection rules, and consistent dashboards without per-source query rewrites.

FLORES targets two schemas:

ECS (Elastic Common Schema) is an open, vendor-neutral field naming standard maintained by Elastic. Critically, ECS is converging with OpenTelemetry (OTel), the CNCF standard for observability instrumentation. This convergence means that ECS-normalized fields will be natively compatible with the OTel ecosystem, giving ECS-aligned data a path into both security and observability pipelines under a single, fully open-source, industry-wide standard.

OCSF (Open Cybersecurity Schema Framework) is a vendor-neutral, open standard developed by a cross-industry consortium including AWS, Splunk, IBM, CrowdStrike, Palo Alto Networks, and others. OCSF defines a full event taxonomy with typed categories and classes.

Together, ECS and OCSF cover the two dominant normalization approaches — field naming and event classification — that security teams are converging on.

FLORES maps FortiOS field names to both, enabling the output CSV datasets to feed directly into SIEM ingestion pipelines, detection rules, and cross-source normalization workflows.

Schema Status Output
ECS (Elastic Common Schema) Active — see coverage below {major}/ECS/{type}_ecs.csv
OCSF Planned {major}/ocsf/

ECS Mapping

Output files

{major}/ECS/ contains one CSV per log category:

File Log category
traffic_ecs.csv Traffic logs (type == traffic)
utm_ecs.csv UTM logs (all security inspection subtypes)
event_ecs.csv Event logs (type == event)

Each file has one row per unique FortiOS field name for that category.

Column Description
Log Field Name FortiOS field identifier (e.g. srcip, proto, logid)
Data Type FortiOS declared type (e.g. ip, string, uint32)
Length Maximum field length from Fortinet docs
Description Field description from Fortinet documentation
Mapping Type See notation below
ECS Field ECS dotted path (e.g. source.ip)
ECS Type ECS field type (e.g. ip, keyword, long)
ECS Description ECS field description

Notation

Mapping type:

  • = — direct 1:1 mapping; value is copied as-is to the ECS field
  • -> — derived or transformed; value is computed, converted, parsed, or aggregated

Multiple ECS targets per FortiOS field use newline-within-cell encoding: each line in the ECS Field, ECS Type, ECS Description, and Mapping Type columns corresponds to one ECS target, in order.

Example — srcip:

Mapping Type:    =          ->                   ->                  ->
ECS Field:       source.ip  network.community_id  network.direction  related.ip
ECS Type:        ip         keyword               keyword            ip
  • source.ip (=): typed IP field enabling range queries
  • network.community_id (->): derived from the full 5-tuple (srcip, dstip, proto, srcport, dstport)
  • network.direction (->): derived by comparing srcip against the observer's known networks
  • related.ip (->): contributes to the event-wide IP array

Example — proto:

Mapping Type:    ->                  ->                 ->
ECS Field:       network.iana_number  network.transport  network.community_id

Example — url:

Mapping Type:    =             ->        ->           ->                        ->
ECS Field:       url.original  url.path  url.query  url.registered_domain  url.top_level_domain

Coverage (FortiOS 7.6)

Log category Total fields Mapped Coverage
Traffic 198 56 ~28%
UTM 262 68 ~26%
Event 390 23 ~6%

Event coverage is low by design — the vast majority of Event log fields are subtype-specific (VPN tunnel details, HA state, system health metrics, etc.) with no general ECS equivalent.

Source files

The first four columns of each *_ecs.csv (Log Field Name, Data Type, Length, Description) come directly from the corresponding {major}/fields/{type}_fields.csv, which is generated by the scraper. After re-scraping for a new FortiOS version, re-run generate_ecs_mappings.py to regenerate the ECS output from the updated field data — the mapping tables in the script apply automatically.

How the mapping is generated

Unlike a manually maintained CSV, the ECS mapping is code-driven. generate_ecs_mappings.py contains:

  1. ECS_FIELD_REGISTRY — a dictionary of all ECS 9.3 fields used by FLORES, including a small number of custom extensions (marked with (*) in their description) for fields that are semantically correct but not yet in the ECS core spec (e.g. network.vrf, session.id, rule.comment).

  2. TRAFFIC_MAPPINGS, UTM_MAPPINGS, EVENT_MAPPINGS — per-category lists of FgtMapping objects that declare which FortiOS field maps to which ECS target(s) and how (= or ->).

  3. main() — iterates over all {major}/fields/ directories, applies the appropriate mapping table, and writes {major}/ECS/{type}_ecs.csv.

To add a new mapping: add a row to the relevant *_MAPPINGS list (and to ECS_FIELD_REGISTRY if the ECS field is new), then re-run the script.

Fields intentionally left unmapped

FortiOS-specific — no ECS equivalent:

  • Virtual domain: vd, vdname
  • Device identity: devname, hostname (when used as device name, not URL hostname)
  • Log metadata: type, subtype, eventtime, itime, srcuuid, dstuuid
  • Policy meta: sessiontype, utmaction, wanoptapptype, appcat, apprisk
  • FortiGuard categories: catdesc, urlsource, urlcat, dlpextra
  • IPS/attack metadata: severity, incidentserialno, crscore when used in certain contexts
  • HA and cluster: haserial, vclustervdom, vclusterid
  • SD-WAN: sla, slaname, linkoutbandwidth, linkinbandwidth
  • Wireless: ap, ssid, radioband, channel, stamac, staintftime
  • VoIP: callduration, callerid, callee, codectype, networkout

Event classification — ECS controlled vocabularies:

ECS event.* fields (event.kind, event.category, event.type, event.outcome) are not free-form strings — they follow a strict controlled-vocabulary specification defined in the ECS category field values reference. Each field accepts only a fixed set of prescribed values with precise semantics.

FortiOS fields like type, subtype, action, and reason carry FortiOS-specific semantics that don't translate cleanly to these prescribed value sets. FortiOS has three top-level log categories (Traffic, UTM, Event) each with multiple subtypes, and each subtype may interpret action or reason differently. Correctly populating ECS event fields requires interpreting per-subtype intent — understanding not just what the field value says, but what it means in context. This kind of interpretation-driven translation introduces inherent bias and is outside the scope of FLORES's field-level schema documentation.

OCSF Mapping

Planned. Will follow the same pattern as ECS mapping once the OCSF field catalog is integrated: a fortigate_ocsf_mapping.csv with per-row structure and =/-> notation.