Skip to content

feat(censys-enrichmentapis): adding new censys-enrichmentapis connector (#7683) - #7688

Open
Krzysztof Retel (kretel) wants to merge 11 commits into
OpenCTI-Platform:masterfrom
kretel:feature/censys-enrichmentapis
Open

Krzysztof Retel (kretel) wants to merge 11 commits into
OpenCTI-Platform:masterfrom
kretel:feature/censys-enrichmentapis

Conversation

@kretel

@kretel Krzysztof Retel (kretel) commented Sep 13, 2026

Copy link
Copy Markdown

Proposed changes

  • Add a new censys-enrichmentapis as internal-enrichment connector.
  • Enrich host (IPv4 and IPv6) using the new host enrichment API, which differs from the host API.

Related issues

Checklist

  • I consider the submitted work as finished
  • I have signed my commits using GPG key.
  • I tested the code for its functionality using different use cases
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

Copilot AI lite review requested due to automatic review settings September 13, 2026 03:09
@filigran-cla-bot filigran-cla-bot Bot added the cla:pending CLA signature required. label Sep 13, 2026
@filigran-cla-bot

Copy link
Copy Markdown

Contributor License Agreement

Hey Krzysztof Retel (@kretel)!

Thank you for your contribution to Filigran! Before we can merge this pull request, we need you to sign our Contributor License Agreement (CLA).

Why do we need a CLA?

The CLA helps protect both you and Filigran. It ensures that:

  • You have the right to make this contribution
  • Filigran can use and distribute your contribution
  • Your contribution remains open source

How to sign

You can sign the CLA using either of these methods:

  1. 🔗 Sign directlyClick here to review and sign the CLA
  2. 📧 Via email — Check your inbox (and spam folder) for a signing invitation from Concord

Once signed, this comment will be automatically updated.


CLA not signed yet

This is an automated message from the Filigran CLA Bot. If you have questions, please contact the maintainers.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Censys EnrichmentAPIs connector for host, domain, and certificate enrichment.

Changes:

  • Adds Censys API clients, converters, and STIX builders.
  • Adds support for services, geography, vulnerabilities, threats, reputation, and certificates.
  • Adds tests, configuration, deployment files, metadata, and documentation.

Reviewed changes

Copilot reviewed 36 out of 40 changed files in this pull request and generated 14 comments.

Show a summary per file
File Review result
internal-enrichment/censys-enrichmentapis/tests/test-requirements.txt No final comments.
internal-enrichment/censys-enrichmentapis/tests/conftest.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_host_enrichment.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_converter.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_connector.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_config.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_builder.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/factories.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/conftest.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/__init__.py No final comments.
internal-enrichment/censys-enrichmentapis/tests/__init__.py No final comments.
internal-enrichment/censys-enrichmentapis/src/requirements.txt No final comments.
internal-enrichment/censys-enrichmentapis/src/main.py No final comments.
internal-enrichment/censys-enrichmentapis/src/config.yml.sample No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/settings.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/errors.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py moderate (3 votes): Service certificates are not converted or related to hosts.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py critical (2 votes): Host results use Host while HostConverter expects HostEnrichment.
moderate (2 votes): Host-derived labels are not propagated to the generated IP observable.
moderate (1 vote): Shared author and marking objects are duplicated across nested conversions.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/certificate.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/base.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/__init__.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/connector.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py moderate (2 votes): threats is omitted from service-field restoration.
critical (1 vote): fetch_hosts() returns Host objects incompatible with HostConverter.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py moderate (1 vote): Missing threats fields can raise AttributeError.
moderate (1 vote): One vulnerability is emitted per CPE instead of per CVE.
moderate (2 votes): Publication dates cause unstable or incorrect Note IDs.
moderate (1 vote): Duplicate deterministic attack patterns and relationships can be emitted.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/network.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/geography.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py moderate (2 votes): Absent CRL values are stored as the literal string "None" at both referenced locations.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py critical (3 votes): Derived data always receives TLP:CLEAR instead of preserving source markings.
moderate (1 vote): Shared context appends duplicate author and marking objects.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/__init__.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builder.py No final comments.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/__init__.py No final comments.
internal-enrichment/censys-enrichmentapis/src/__init__.py No final comments.
internal-enrichment/censys-enrichmentapis/README.md nit (3 votes): Manual deployment references the wrong sample configuration filename.
nit (2 votes): The documented command resolves to a nonexistent src/src/main.py.
nit (3 votes): The reputation example does not match the emitted note format.
internal-enrichment/censys-enrichmentapis/Dockerfile No final comments.
internal-enrichment/censys-enrichmentapis/docker-compose.yml moderate (3 votes): CENSYS_ENRICHMENT_MAX_TLP is commented out.
critical (1 vote): The compose image has an invalid namespace/name and directory typo.
internal-enrichment/censys-enrichmentapis/.gitignore No final comments.
internal-enrichment/censys-enrichmentapis/__metadata__/connector_manifest.json critical (3 votes): The image name contains a directory typo and fails metadata validation and Docker Hub discovery.
internal-enrichment/censys-enrichmentapis/__metadata__/connector_config_schema.json No final comments.
internal-enrichment/censys-enrichmentapis/__metadata__/CONNECTOR_CONFIG_DOC.md No final comments.
Suppressed comments (7)

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py:25

  • This shared context is reused by the domain converter for each resolved host and again for certificates, so repeated calls append the same author and marking objects multiple times to one STIX bundle. Make this method idempotent to avoid duplicate object IDs in domain enrichments.
        self.bundle.extend([self.author, self.marking])

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py:52

  • As with CRL distribution points above, an absent policy list becomes the literal string "None" in the STIX object. Guard the assignment so missing certificate policies remain None.
        certificate.certificate_policies = str(
            cert.parsed.extensions.certificate_policies
        )

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:296

  • HostEnrichmentService responses can omit the threats field (the tests inject it dynamically), so direct attribute access raises AttributeError for ordinary services without threats. Use the same tolerant accessor as the other threat paths.
        for threat in service.threats or []:

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:145

  • A single CVE can contain evidence for multiple CPEs, but this loop calls add_vulnerability once per CPE. That emits the same deterministic Vulnerability ID multiple times in one bundle (and duplicates its relationships); create one vulnerability per CVE and link it to each affected software instead.
            for vulnerability in self._get_value(service, "vulns") or []:
                for cpe in self._vulnerability_cpes(vulnerability):
                    software = software_by_cpe.get(cpe)
                    if software is None:
                        software = self._add_software_from_cpe(observable, cpe)

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:337

  • Passing scan_time as publication_date sets Note.created; connectors_sdk.Note.to_stix2_object() then passes that datetime to PyctiNote.generate_id. This violates the repository's deterministic-note rule and makes scan timestamps part of the note identity; leave publication_date unset and keep scan time only in the content.
                    publication_date=datetime.datetime.fromisoformat(
                        service.scan_time
                    ),
                    authors=[self._context.author.name],

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:399

  • A fresh deterministic AttackPattern is appended for every threat tactic. If two threats share a tactic, the same name/ATT&CK ID yields the same STIX ID, so the bundle contains duplicate attack-pattern objects and duplicate relationships. Cache patterns and relationships by deterministic ID before appending them.
            for tactic in self._get_value(threat, "tactic") or []:
                attack_pattern = self._add_threat_attack_pattern(tactic)
                if attack_pattern:
                    self.bundle.extend(
                        [
                            attack_pattern,
                            Relationship(
                                source=observable,
                                target=attack_pattern,
                                type=RelationshipType.RELATED_TO,
                                **self.common_props,
                            ),
                        ]
                    )

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:47

  • For domain enrichment, each host conversion already calls add_author_and_marking(), and this method appends the same deterministic identity and marking again after all hosts. A domain with multiple hosts therefore sends repeated objects with identical STIX IDs (and repeats them again per certificate), unnecessarily bloating the bundle and risking duplicate-object rejection. Add these shared context objects once per bundle, not once per nested conversion.
        self.builder.add_author_and_marking()

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal-enrichment/censys-enrichmentapis/__metadata__/connector_manifest.json Outdated
Comment thread internal-enrichment/censys-enrichmentapis/docker-compose.yml Outdated
Comment on lines +16 to +17
self.marking = TLPMarking(level=TLPLevel.CLEAR)
self.common_props = {"author": self.author, "markings": [self.marking]}
if hit.certificate_v1:
yield hit.certificate_v1.resource

def fetch_hosts(self, hostname: str) -> Generator[Host, None, None]:
Comment on lines +89 to +92
self.builder.services.add_service_vulnerabilities(
observable=observable,
services=data.services,
)
Comment thread internal-enrichment/censys-enrichmentapis/README.md Outdated
Comment thread internal-enrichment/censys-enrichmentapis/README.md Outdated
Comment thread internal-enrichment/censys-enrichmentapis/README.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 36 out of 40 changed files in this pull request and generated 1 comment.

Suppressed comments (10)

Previously missed (1) — in code that hasn't changed since the last review.

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:336

  • publication_date is deprecated by the SDK and is converted to created, which participates in deterministic Note ID generation. Using the volatile service scan timestamp here creates a new Note ID whenever Censys rescans the service, causing duplicate notes to accumulate; do not use scan_time as the Note creation timestamp and choose a stable identity strategy for the displayed scan time.

internal-enrichment/censys-enrichmentapis/README.md:112

  • The documented sample filename is wrong: this connector ships src/config.yml.sample, not src/config.yaml.sample. Following the manual deployment instructions therefore fails at the first copy step.
1. Copy `src/config.yaml.sample` to `src/config.yml` and configure with your credentials.

internal-enrichment/censys-enrichmentapis/docker-compose.yml:21

  • This environment variable is commented out, so setting CENSYS_ENRICHMENT_MAX_TLP in .env has no effect when operators use the provided compose file, despite the README and generated schema documenting it. Pass the variable through to the container with its documented default.
      # - CENSYS_ENRICHMENT_MAX_TLP=${CENSYS_ENRICHMENT_MAX_TLP:-TLP:AMBER}

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py:16

  • All generated objects use a hard-coded TLP:CLEAR marking, even when _process has accepted an input marked TLP:AMBER or a stricter level allowed by configuration. This downgrades derived enrichment data and can expose restricted information; pass through the source observable's marking references and use custom TLP:CLEAR only when the source is unmarked.
        self.marking = TLPMarking(level=TLPLevel.CLEAR)

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:414

  • censys-platform deserializes threat.malware as a ThreatMalware model, not a dict, so this guard drops malware for real API responses; the tests only exercise dict fixtures. Read the fields through the existing _get_value helper (or normalize with model_dump) so both SDK models and dict test data are handled.
        if not isinstance(malware_data, dict):
            return None

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:517

  • The same SDK-model-versus-dict mismatch causes the malware details to be omitted from every real threat note: ThreatMalware cannot pass this isinstance(..., dict) check. Use _get_value for primary_name, all_names, and last_updated_at, as in the rest of this builder.
            if isinstance(malware_data, dict) and malware_data.get("primary_name"):

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:514

  • Threat evidence is also deserialized as SDK Evidence objects, but this branch only renders dict evidence. Consequently the real API's data_path and found_value are silently absent from threat notes even though the tests pass dicts; use _get_value for these fields.
                    if isinstance(item, dict):
                        data_path = item.get("data_path", "unknown")
                        found_value = item.get("found_value", "")
                        content_parts.append(f"- {data_path}: {found_value}")

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:30

  • fetch_hosts() returns the Host model from the Search API, but this call passes those objects into HostConverter, which assumes the HostEnrichment shape (labels, reputation, and enrichment-service fields). The real domain path can therefore raise AttributeError before producing a bundle; the tests mask this by using HostEnrichment instances as the search results. Normalize the search response or make the converter handle both model shapes.
            host_converter._convert(observable=ip_stix.to_stix2_object(), data=host)

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:30

  • Because the nested HostConverter is invoked directly here, its primary_observable_labels are never copied to DomainConverter.primary_observable_labels, which is the list the connector merges into the original observable. Labels derived from hosts resolved for a domain are therefore silently lost; propagate them or explicitly apply them to the generated IP observable.
            host_converter._convert(observable=ip_stix.to_stix2_object(), data=host)

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py:82

  • The host conversion never processes service.cert through CertificateStixBuilder, so TLS certificates returned with service data are dropped even though the connector documentation maps services.cert to an X509-Certificate. Add the service-certificate conversion (and restore the raw field if the generated API model omits it) or remove the documented behavior.
        self.builder.services.add_service_notes(
            observable=observable,
            services=data.services,
        )

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 36 out of 40 changed files in this pull request and generated 1 comment.

Suppressed comments (14)

Previously missed (1) — in code that hasn't changed since the last review.

internal-enrichment/censys-enrichmentapis/README.md:112

  • The manual deployment step points to src/config.yaml.sample, but the file added by this connector is src/config.yml.sample; following the documented command therefore fails because the referenced file does not exist.

internal-enrichment/censys-enrichmentapis/README.md:294

  • The emitted reputation note uses - Label: ... in add_reputation_note, but this section documents the same field as Score Level and shows - Score level: .... Please align the prose and example with the actual note format so operators can parse the enrichment output correctly.
- **Score Level**: Risk classification (LOW, MEDIUM_RISK, HIGH, CRITICAL)
- **Model Version**: Version of the reputation model used

internal-enrichment/censys-enrichmentapis/docker-compose.yml:7

  • The compose file tags the image without the opencti/ namespace, while the manifest, README, and the other connector compose files use opencti/connector-censys-enrichmentapis. This breaks workflows that expect the compose image name to match the published container image. Use the published image name here.
    image: connector-censys-enrichmentapis:latest

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py:17

  • These common properties always use a TLP:CLEAR marking, but _process accepts inputs up to the configured maximum (TLP:AMBER by default) and never passes the source marking into the converter. Enriching an AMBER observable therefore emits all derived objects as CLEAR, downgrading their access controls. Build the context from the source objectMarking and fall back to the custom CLEAR marking only when the source is unmarked.
        self.author = OrganizationAuthor(name="Censys EnrichmentAPIs Connector")
        self.marking = TLPMarking(level=TLPLevel.CLEAR)
        self.common_props = {"author": self.author, "markings": [self.marking]}

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py:44

  • When the parsed extensions object exists but Censys omits crl_distribution_points, this assigns the literal string "None" to the X509 field, falsely claiming a CRL value. Guard the optional field before converting it to a string; apply the same treatment to certificate_policies below.
        certificate.crl_distribution_points = str(
            cert.parsed.extensions.crl_distribution_points
        )

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py:52

  • As with crl_distribution_points above, an absent optional policy list is serialized as the literal string "None" instead of being omitted, so certificates with partial metadata contain inaccurate extension data. Guard this value before calling str.
        certificate.certificate_policies = str(
            cert.parsed.extensions.certificate_policies
        )

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:371

  • Note.to_stix2_object() feeds created into Note.generate_id; using the Censys scan_time here makes every refreshed scan produce a new ID for the same service. This creates unbounded duplicate notes and queue churn instead of updating the existing enrichment. Keep the scan timestamp in the rendered content, but use a stable note identity that does not include a datetime.
                    created=datetime.datetime.fromisoformat(
                        scan_time
                    ),

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:213

  • This helper also converts the publication timestamp into Note.created, and the SDK includes that datetime in Note.generate_id. A repeated banner observed at a new timestamp therefore becomes a new note instead of an update; keep the date for display but use a stable identity seed that excludes it.
                created=datetime.datetime.fromisoformat(publication_date),

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py:109

  • The search response exposes next_page_token, but this generator yields only the first page. An X509 lookup with more results than the API page size will silently omit certificates. Iterate through all page tokens before completing the generator, as the other search-based methods must also do.
            if res.result.result:
                for hit in res.result.result.hits:
                    if hit.certificate_v1:
                        yield hit.certificate_v1.resource

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py:130

  • Domain host discovery stops after the first res.result.result.hits page even though the response carries next_page_token. Domains with more resolved IPs than one API page will therefore be partially enriched, contrary to the documented domain behavior. Loop through the search pages before yielding all hosts.
            res: V3GlobaldataSearchQueryResponse = sdk.global_data.search(
                search_query_input_body=search_query
            )
            if res.result.result:
                for hit in res.result.result.hits:
                    if hit.host_v1:
                        yield hit.host_v1.resource

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py:153

  • Certificate discovery by domain also consumes only the first search page despite the response's next_page_token. A domain with many matching certificates will silently lose the remaining results. Iterate through all pages before yielding certificates.
            res: V3GlobaldataSearchQueryResponse = sdk.global_data.search(
                search_query_input_body=search_query
            )
            if res.result.result:
                for hit in res.result.result.hits:
                    if hit.certificate_v1:
                        yield hit.certificate_v1.resource

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:30

  • HostConverter._convert stores host labels in host_converter.primary_observable_labels, but this nested call ignores that list. For domain enrichment, the generated IP observables therefore lose Censys service, threat, and reputation labels, and the top-level domain cannot merge them either. Propagate the labels to each generated ip_stix before serializing the bundle.
        for host in hosts:
            ip_stix = self.builder.network.add_ip(
                observable=Reference(id=stix_entity.get("id")),
                ip=host.ip,
            )
            host_converter._convert(observable=ip_stix.to_stix2_object(), data=host)

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py:91

  • HostEnrichmentService includes a cert, but this conversion only emits service notes, vulnerabilities, and threats; it never converts per-service certificates or relates them to the IP. IP enrichment therefore drops all TLS certificates, despite the connector documentation mapping services.cert and the existing Censys connector handling them in internal-enrichment/censys-enrichment/src/censys_enrichment/builder.py:361-373. Add the certificate conversion and relationship for each service.
        self.builder.services.add_service_notes(
            observable=observable,
            services=data.services,
        )

internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py:49

  • This converter is also invoked once for every resolved host by DomainConverter._append_hosts; each invocation appends the same deterministic author and marking, and _append_domain_certs appends them again. A domain with multiple hosts consequently returns duplicate identity/marking objects with the same IDs, unnecessarily inflating the bundle. Add shared metadata once per top-level conversion or guard the insertion.
        self.builder.add_author_and_marking()

Comment on lines +24 to +25
def add_author_and_marking(self) -> None:
self.bundle.extend([self.author, self.marking])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:pending CLA signature required.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants