feat(censys-enrichmentapis): adding new censys-enrichmentapis connector (#7683) - #7688
Krzysztof Retel (kretel) wants to merge 11 commits into
Conversation
The template was generated following the documentation.
…n key/value table.
Contributor License AgreementHey Krzysztof Retel (@kretel)! Thank you for your contribution to Filigran! Before we can merge this pull request, we need you to sign our Contributor License Agreement (CLA). Why do we need a CLA?The CLA helps protect both you and Filigran. It ensures that:
How to signYou can sign the CLA using either of these methods:
Once signed, this comment will be automatically updated. ❌ CLA not signed yet This is an automated message from the Filigran CLA Bot. If you have questions, please contact the maintainers. |
There was a problem hiding this comment.
Pull request overview
Adds a Censys EnrichmentAPIs connector for host, domain, and certificate enrichment.
Changes:
- Adds Censys API clients, converters, and STIX builders.
- Adds support for services, geography, vulnerabilities, threats, reputation, and certificates.
- Adds tests, configuration, deployment files, metadata, and documentation.
Reviewed changes
Copilot reviewed 36 out of 40 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Review result |
|---|---|
internal-enrichment/censys-enrichmentapis/tests/test-requirements.txt |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/conftest.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_host_enrichment.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_converter.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_connector.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_config.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/test_builder.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/factories.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/conftest.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/censys_enrichmentapis/__init__.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/tests/__init__.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/requirements.txt |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/main.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/config.yml.sample |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/settings.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/errors.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py |
moderate (3 votes): Service certificates are not converted or related to hosts. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py |
critical (2 votes): Host results use Host while HostConverter expects HostEnrichment.moderate (2 votes): Host-derived labels are not propagated to the generated IP observable. moderate (1 vote): Shared author and marking objects are duplicated across nested conversions. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/certificate.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/base.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/__init__.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/connector.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py |
moderate (2 votes): threats is omitted from service-field restoration.critical (1 vote): fetch_hosts() returns Host objects incompatible with HostConverter. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py |
moderate (1 vote): Missing threats fields can raise AttributeError.moderate (1 vote): One vulnerability is emitted per CPE instead of per CVE. moderate (2 votes): Publication dates cause unstable or incorrect Note IDs. moderate (1 vote): Duplicate deterministic attack patterns and relationships can be emitted. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/network.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/geography.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py |
moderate (2 votes): Absent CRL values are stored as the literal string "None" at both referenced locations. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py |
critical (3 votes): Derived data always receives TLP:CLEAR instead of preserving source markings. moderate (1 vote): Shared context appends duplicate author and marking objects. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/__init__.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builder.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/__init__.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/src/__init__.py |
No final comments. |
internal-enrichment/censys-enrichmentapis/README.md |
nit (3 votes): Manual deployment references the wrong sample configuration filename. nit (2 votes): The documented command resolves to a nonexistent src/src/main.py.nit (3 votes): The reputation example does not match the emitted note format. |
internal-enrichment/censys-enrichmentapis/Dockerfile |
No final comments. |
internal-enrichment/censys-enrichmentapis/docker-compose.yml |
moderate (3 votes): CENSYS_ENRICHMENT_MAX_TLP is commented out.critical (1 vote): The compose image has an invalid namespace/name and directory typo. |
internal-enrichment/censys-enrichmentapis/.gitignore |
No final comments. |
internal-enrichment/censys-enrichmentapis/__metadata__/connector_manifest.json |
critical (3 votes): The image name contains a directory typo and fails metadata validation and Docker Hub discovery. |
internal-enrichment/censys-enrichmentapis/__metadata__/connector_config_schema.json |
No final comments. |
internal-enrichment/censys-enrichmentapis/__metadata__/CONNECTOR_CONFIG_DOC.md |
No final comments. |
Suppressed comments (7)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py:25
- This shared context is reused by the domain converter for each resolved host and again for certificates, so repeated calls append the same author and marking objects multiple times to one STIX bundle. Make this method idempotent to avoid duplicate object IDs in domain enrichments.
self.bundle.extend([self.author, self.marking])
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py:52
- As with CRL distribution points above, an absent policy list becomes the literal string
"None"in the STIX object. Guard the assignment so missing certificate policies remainNone.
certificate.certificate_policies = str(
cert.parsed.extensions.certificate_policies
)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:296
HostEnrichmentServiceresponses can omit thethreatsfield (the tests inject it dynamically), so direct attribute access raisesAttributeErrorfor ordinary services without threats. Use the same tolerant accessor as the other threat paths.
for threat in service.threats or []:
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:145
- A single CVE can contain evidence for multiple CPEs, but this loop calls
add_vulnerabilityonce per CPE. That emits the same deterministic Vulnerability ID multiple times in one bundle (and duplicates its relationships); create one vulnerability per CVE and link it to each affected software instead.
for vulnerability in self._get_value(service, "vulns") or []:
for cpe in self._vulnerability_cpes(vulnerability):
software = software_by_cpe.get(cpe)
if software is None:
software = self._add_software_from_cpe(observable, cpe)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:337
- Passing
scan_timeaspublication_datesetsNote.created;connectors_sdk.Note.to_stix2_object()then passes that datetime toPyctiNote.generate_id. This violates the repository's deterministic-note rule and makes scan timestamps part of the note identity; leavepublication_dateunset and keep scan time only in the content.
publication_date=datetime.datetime.fromisoformat(
service.scan_time
),
authors=[self._context.author.name],
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:399
- A fresh deterministic
AttackPatternis appended for every threat tactic. If two threats share a tactic, the same name/ATT&CK ID yields the same STIX ID, so the bundle contains duplicate attack-pattern objects and duplicate relationships. Cache patterns and relationships by deterministic ID before appending them.
for tactic in self._get_value(threat, "tactic") or []:
attack_pattern = self._add_threat_attack_pattern(tactic)
if attack_pattern:
self.bundle.extend(
[
attack_pattern,
Relationship(
source=observable,
target=attack_pattern,
type=RelationshipType.RELATED_TO,
**self.common_props,
),
]
)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:47
- For domain enrichment, each host conversion already calls
add_author_and_marking(), and this method appends the same deterministic identity and marking again after all hosts. A domain with multiple hosts therefore sends repeated objects with identical STIX IDs (and repeats them again per certificate), unnecessarily bloating the bundle and risking duplicate-object rejection. Add these shared context objects once per bundle, not once per nested conversion.
self.builder.add_author_and_marking()
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| self.marking = TLPMarking(level=TLPLevel.CLEAR) | ||
| self.common_props = {"author": self.author, "markings": [self.marking]} |
| if hit.certificate_v1: | ||
| yield hit.certificate_v1.resource | ||
|
|
||
| def fetch_hosts(self, hostname: str) -> Generator[Host, None, None]: |
| self.builder.services.add_service_vulnerabilities( | ||
| observable=observable, | ||
| services=data.services, | ||
| ) |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 40 changed files in this pull request and generated 1 comment.
Suppressed comments (10)
Previously missed (1) — in code that hasn't changed since the last review.
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:336
publication_dateis deprecated by the SDK and is converted tocreated, which participates in deterministic Note ID generation. Using the volatile service scan timestamp here creates a new Note ID whenever Censys rescans the service, causing duplicate notes to accumulate; do not usescan_timeas the Note creation timestamp and choose a stable identity strategy for the displayed scan time.
internal-enrichment/censys-enrichmentapis/README.md:112
- The documented sample filename is wrong: this connector ships
src/config.yml.sample, notsrc/config.yaml.sample. Following the manual deployment instructions therefore fails at the first copy step.
1. Copy `src/config.yaml.sample` to `src/config.yml` and configure with your credentials.
internal-enrichment/censys-enrichmentapis/docker-compose.yml:21
- This environment variable is commented out, so setting
CENSYS_ENRICHMENT_MAX_TLPin.envhas no effect when operators use the provided compose file, despite the README and generated schema documenting it. Pass the variable through to the container with its documented default.
# - CENSYS_ENRICHMENT_MAX_TLP=${CENSYS_ENRICHMENT_MAX_TLP:-TLP:AMBER}
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py:16
- All generated objects use a hard-coded TLP:CLEAR marking, even when
_processhas accepted an input marked TLP:AMBER or a stricter level allowed by configuration. This downgrades derived enrichment data and can expose restricted information; pass through the source observable's marking references and use custom TLP:CLEAR only when the source is unmarked.
self.marking = TLPMarking(level=TLPLevel.CLEAR)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:414
censys-platformdeserializesthreat.malwareas aThreatMalwaremodel, not a dict, so this guard drops malware for real API responses; the tests only exercise dict fixtures. Read the fields through the existing_get_valuehelper (or normalize withmodel_dump) so both SDK models and dict test data are handled.
if not isinstance(malware_data, dict):
return None
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:517
- The same SDK-model-versus-dict mismatch causes the malware details to be omitted from every real threat note:
ThreatMalwarecannot pass thisisinstance(..., dict)check. Use_get_valueforprimary_name,all_names, andlast_updated_at, as in the rest of this builder.
if isinstance(malware_data, dict) and malware_data.get("primary_name"):
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:514
- Threat evidence is also deserialized as SDK
Evidenceobjects, but this branch only renders dict evidence. Consequently the real API'sdata_pathandfound_valueare silently absent from threat notes even though the tests pass dicts; use_get_valuefor these fields.
if isinstance(item, dict):
data_path = item.get("data_path", "unknown")
found_value = item.get("found_value", "")
content_parts.append(f"- {data_path}: {found_value}")
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:30
fetch_hosts()returns theHostmodel from the Search API, but this call passes those objects intoHostConverter, which assumes theHostEnrichmentshape (labels,reputation, and enrichment-service fields). The real domain path can therefore raiseAttributeErrorbefore producing a bundle; the tests mask this by usingHostEnrichmentinstances as the search results. Normalize the search response or make the converter handle both model shapes.
host_converter._convert(observable=ip_stix.to_stix2_object(), data=host)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:30
- Because the nested
HostConverteris invoked directly here, itsprimary_observable_labelsare never copied toDomainConverter.primary_observable_labels, which is the list the connector merges into the original observable. Labels derived from hosts resolved for a domain are therefore silently lost; propagate them or explicitly apply them to the generated IP observable.
host_converter._convert(observable=ip_stix.to_stix2_object(), data=host)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py:82
- The host conversion never processes
service.certthroughCertificateStixBuilder, so TLS certificates returned with service data are dropped even though the connector documentation mapsservices.certto an X509-Certificate. Add the service-certificate conversion (and restore the raw field if the generated API model omits it) or remove the documented behavior.
self.builder.services.add_service_notes(
observable=observable,
services=data.services,
)
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 40 changed files in this pull request and generated 1 comment.
Suppressed comments (14)
Previously missed (1) — in code that hasn't changed since the last review.
internal-enrichment/censys-enrichmentapis/README.md:112
- The manual deployment step points to
src/config.yaml.sample, but the file added by this connector issrc/config.yml.sample; following the documented command therefore fails because the referenced file does not exist.
internal-enrichment/censys-enrichmentapis/README.md:294
- The emitted reputation note uses
- Label: ...inadd_reputation_note, but this section documents the same field asScore Leveland shows- Score level: .... Please align the prose and example with the actual note format so operators can parse the enrichment output correctly.
- **Score Level**: Risk classification (LOW, MEDIUM_RISK, HIGH, CRITICAL)
- **Model Version**: Version of the reputation model used
internal-enrichment/censys-enrichmentapis/docker-compose.yml:7
- The compose file tags the image without the
opencti/namespace, while the manifest, README, and the other connector compose files useopencti/connector-censys-enrichmentapis. This breaks workflows that expect the compose image name to match the published container image. Use the published image name here.
image: connector-censys-enrichmentapis:latest
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/base.py:17
- These common properties always use a TLP:CLEAR marking, but
_processaccepts inputs up to the configured maximum (TLP:AMBER by default) and never passes the source marking into the converter. Enriching an AMBER observable therefore emits all derived objects as CLEAR, downgrading their access controls. Build the context from the sourceobjectMarkingand fall back to the custom CLEAR marking only when the source is unmarked.
self.author = OrganizationAuthor(name="Censys EnrichmentAPIs Connector")
self.marking = TLPMarking(level=TLPLevel.CLEAR)
self.common_props = {"author": self.author, "markings": [self.marking]}
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py:44
- When the parsed extensions object exists but Censys omits
crl_distribution_points, this assigns the literal string"None"to the X509 field, falsely claiming a CRL value. Guard the optional field before converting it to a string; apply the same treatment tocertificate_policiesbelow.
certificate.crl_distribution_points = str(
cert.parsed.extensions.crl_distribution_points
)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/certificate.py:52
- As with
crl_distribution_pointsabove, an absent optional policy list is serialized as the literal string"None"instead of being omitted, so certificates with partial metadata contain inaccurate extension data. Guard this value before callingstr.
certificate.certificate_policies = str(
cert.parsed.extensions.certificate_policies
)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:371
Note.to_stix2_object()feedscreatedintoNote.generate_id; using the Censysscan_timehere makes every refreshed scan produce a new ID for the same service. This creates unbounded duplicate notes and queue churn instead of updating the existing enrichment. Keep the scan timestamp in the rendered content, but use a stable note identity that does not include a datetime.
created=datetime.datetime.fromisoformat(
scan_time
),
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/builders/service.py:213
- This helper also converts the publication timestamp into
Note.created, and the SDK includes that datetime inNote.generate_id. A repeated banner observed at a new timestamp therefore becomes a new note instead of an update; keep the date for display but use a stable identity seed that excludes it.
created=datetime.datetime.fromisoformat(publication_date),
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py:109
- The search response exposes
next_page_token, but this generator yields only the first page. An X509 lookup with more results than the API page size will silently omit certificates. Iterate through all page tokens before completing the generator, as the other search-based methods must also do.
if res.result.result:
for hit in res.result.result.hits:
if hit.certificate_v1:
yield hit.certificate_v1.resource
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py:130
- Domain host discovery stops after the first
res.result.result.hitspage even though the response carriesnext_page_token. Domains with more resolved IPs than one API page will therefore be partially enriched, contrary to the documented domain behavior. Loop through the search pages before yielding all hosts.
res: V3GlobaldataSearchQueryResponse = sdk.global_data.search(
search_query_input_body=search_query
)
if res.result.result:
for hit in res.result.result.hits:
if hit.host_v1:
yield hit.host_v1.resource
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/client.py:153
- Certificate discovery by domain also consumes only the first search page despite the response's
next_page_token. A domain with many matching certificates will silently lose the remaining results. Iterate through all pages before yielding certificates.
res: V3GlobaldataSearchQueryResponse = sdk.global_data.search(
search_query_input_body=search_query
)
if res.result.result:
for hit in res.result.result.hits:
if hit.certificate_v1:
yield hit.certificate_v1.resource
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/domain.py:30
HostConverter._convertstores host labels inhost_converter.primary_observable_labels, but this nested call ignores that list. For domain enrichment, the generated IP observables therefore lose Censys service, threat, and reputation labels, and the top-level domain cannot merge them either. Propagate the labels to each generatedip_stixbefore serializing the bundle.
for host in hosts:
ip_stix = self.builder.network.add_ip(
observable=Reference(id=stix_entity.get("id")),
ip=host.ip,
)
host_converter._convert(observable=ip_stix.to_stix2_object(), data=host)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py:91
HostEnrichmentServiceincludes acert, but this conversion only emits service notes, vulnerabilities, and threats; it never converts per-service certificates or relates them to the IP. IP enrichment therefore drops all TLS certificates, despite the connector documentation mappingservices.certand the existing Censys connector handling them ininternal-enrichment/censys-enrichment/src/censys_enrichment/builder.py:361-373. Add the certificate conversion and relationship for each service.
self.builder.services.add_service_notes(
observable=observable,
services=data.services,
)
internal-enrichment/censys-enrichmentapis/src/censys_enrichmentapis/converters/host.py:49
- This converter is also invoked once for every resolved host by
DomainConverter._append_hosts; each invocation appends the same deterministic author and marking, and_append_domain_certsappends them again. A domain with multiple hosts consequently returns duplicate identity/marking objects with the same IDs, unnecessarily inflating the bundle. Add shared metadata once per top-level conversion or guard the insertion.
self.builder.add_author_and_marking()
| def add_author_and_marking(self) -> None: | ||
| self.bundle.extend([self.author, self.marking]) |
Proposed changes
Related issues
Checklist
Further comments