Skip to content

Commit 624d094

Browse files
authored
Doc fixes (#914)
* concepts/metric_hub.md: List bigquery-etl, Python and Looker as Metric Hub consumers Finding-Id: ad42a11ed20841fca245ae0a967b8e69 Source: https://github.com/mozilla/metric-hub * concepts/pipeline/gcp_data_pipeline.md: Clarify Live Sink is a configured ingestion-sink deployment Finding-Id: de42853fecc94c469890cbaa23b7e3b6 Source: https://github.com/mozilla/gcp-ingestion/blob/main/README.md Source: https://github.com/mozilla/gcp-ingestion/blob/main/docs/architecture/bigquery_sink_specification.md * concepts/pipeline/http_edge_spec.md: State that the edge forwards submissions to Google Cloud PubSub Finding-Id: 3e2f53049e414bbbbf7b49b5a307b4d6 Source: https://github.com/mozilla/gcp-ingestion/blob/main/docs/architecture/edge_service_specification.md Source: https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-edge/ingestion_edge/publish.py * cookbooks/deploying-containers.md: Add note that GCR is retired and point to the GAR guide Finding-Id: f6da44cec07c46319879d4e6cca4594d Source: https://mozilla-hub.atlassian.net/wiki/spaces/SRE/pages/997163545/How+to+Publish+Container+Images+to+GAR * cookbooks/publishing_datasets.md: Use list-valued review_bugs in metadata.yaml example Finding-Id: c87181306ab247b98733ede6f727eaea Source: https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/metadata/validate_metadata.py Source: https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/telemetry_derived/ssl_ratios_v1/metadata.yaml Source: https://mozilla.github.io/bigquery-etl/reference/public_data/ * datasets/glam.md: Note shared-prod glam_etl tables are views of moz-fx-glam-prod Finding-Id: 46786d63a92846638b4d0d4b9a7980e2 Source: https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/glam_etl/dataset_metadata.yaml Source: https://github.com/mozilla/bigquery-etl/tree/HEAD/sql/moz-fx-glam-prod/glam_etl * datasets/other/sponsored_tiles/sponsored_tiles.md: Mark event_aggregates_spons_tiles as being deprecated Finding-Id: 02f4aae979614a37b0c57a57d7d50c5c Source: https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/contextual_services_derived/event_aggregates_spons_tiles_v1/metadata.yaml * datasets/pings.md: Note that Glean's events ping is sent on background/event limit Finding-Id: 37bfab48aa3b4aefabc68b2aa1537ba7 Source: https://mozilla.github.io/glean/book/user/pings/sent-by-glean.html * introduction/tools.md: Note that deprecated tools are no longer available Finding-Id: 3ab14bab44e645c18fc7f76d185980a5 Source: https://mozilla.github.io/telemetry-dashboard/new-pipeline/evo.html * metrics/metrics.md: Mark Day 2-7 Activation OKR status as historical (2020) Finding-Id: f374eab960af44b59701556702d8da4e * metrics/metrics.md: Stop calling Confluence Metrics Inventory the canonical source Finding-Id: b1b4fde5eeca4ae68db8dddaf57bfcba * metrics/usage.md: Point readers to canonical usage definition sources Finding-Id: 9b9fbf55d733405e9e21e718ef75aa44 Source: https://docs.telemetry.mozilla.org/metrics/metrics Source: https://github.com/mozilla/bigquery-etl/tree/HEAD/sql/moz-fx-data-shared-prod/telemetry/active_users_aggregates Source: https://github.com/mozilla/metric-hub/tree/HEAD/definitions Source: https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/314704478/Daily+Active+Users+DAU+Metric * tools/guiding_principles.md: List Looker ahead of Redash as the primary data access tool Finding-Id: 034c94f1eabe49fc913d8b9bb822a807 Source: https://docs.telemetry.mozilla.org/tools/guiding_principles.html * tools/projects.md: Mark python_moztelemetry as archived Finding-Id: 51ee7e5f0e094a0bbfe99d846428bfd6 Source: https://github.com/mozilla/python_moztelemetry/blob/master/README.md * tools/projects.md: Mark moztelemetry as archived Finding-Id: e5a61b30903c4d6398daefd7031de2a7 Source: https://github.com/mozilla/moztelemetry/blob/main/README.md * tools/projects.md: Mark telemetry-batch-view as legacy Finding-Id: 0e04a140e0094970a79ac87789a9a168 Source: https://github.com/mozilla/telemetry-batch-view/blob/main/README.md * tools/projects.md: Mark telemetry-streaming as no longer in use Finding-Id: 865a2c26a3584f6e818d627aee31ea9a Source: https://github.com/mozilla/telemetry-streaming/blob/master/README.md * datasets/pings.md: Fix dead coverage ping link The coverage ping was removed from Firefox in bug 1773060 and its source doc no longer exists. Link to the last version of the doc and note that the ping is no longer sent.
1 parent a4db25f commit 624d094

13 files changed

Lines changed: 39 additions & 28 deletions

File tree

‎src/concepts/metric_hub.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Metric Hub is a [repository](https://github.com/mozilla/metric-hub) that contains metric, data source and segment definitions that have been
44
reviewed and can be seen as the source of truth.
5-
Definitions that are part of Metric Hub can be referenced in configurations of other tooling as well, such as [Jetstream](https://experimenter.info/data-analysis/jetstream/overview) and [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html?highlight=opmon#operational-monitoring-opmon).
5+
Definitions that are part of Metric Hub can be referenced in configurations of other tooling as well, such as [Jetstream](https://experimenter.info/data-analysis/jetstream/overview), [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html?highlight=opmon#operational-monitoring-opmon), [bigquery-etl](https://github.com/mozilla/bigquery-etl), Python scripts and [Looker](https://mozilla.cloud.looker.com) explores.
66

77
Metric Hub is a "thin" metric layer that lies between the Data Warehouse and consumers, such as analysis tools.
88
The Data Warehouse contains the datasets and tables referenced in the metric definitions. Analysis tools can reference metrics that are defined in metric-hub.

‎src/concepts/pipeline/gcp_data_pipeline.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,8 +131,8 @@ and the [schema generator].
131131
The result are tables that contains SQL-friendly field names for all known
132132
measures, as implemented in the [probe scraper].
133133

134-
A Kubernetes `Live Sink` job (part of the [`ingestion-sink`][ingestion-sink]
135-
service) reads from the Decoded topic and writes out to
134+
A Kubernetes `Live Sink` job (a deployment of the [`ingestion-sink`][ingestion-sink]
135+
Java application, configured for this role) reads from the Decoded topic and writes out to
136136
**[live ping tables][table layout]**.
137137
These tables are updated frequently, and typically reflect data within a few
138138
minutes of it being ingested. They are intended for low-latency access to

‎src/concepts/pipeline/http_edge_spec.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ The original implementation of the HTTP Edge Server was tracked in
1111
HTTP submissions come in from the wild, hit a load balancer,
1212
then the HTTP Edge Server described in this document.
1313
Data is accepted via a POST/PUT request from clients,
14-
and forwarded to the [Data Pipeline](gcp_data_pipeline.md), where
14+
wrapped in a PubSub message, and forwarded to Google Cloud PubSub for the
15+
[Data Pipeline](gcp_data_pipeline.md), where
1516
any further processing, analysis, and storage will be handled.
1617

1718
Submission payloads are expected to be optionally-gzipped JSON

‎src/cookbooks/deploying-containers.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
The following cookbook describes how to set up automated build and deployment for containers with CircleCI, a useful pattern for scheduling custom jobs in Google Kubernetes Engine.
44

5+
> **Note:** Google Container Registry (GCR) has been shut down, so this cookbook is outdated. Google Artifact Registry (GAR) is now the standard, required path for MozCloud services deploying containers via ArgoCD; see [How to Publish Container Images to GAR](https://mozilla-hub.atlassian.net/wiki/spaces/SRE/pages/997163545/How+to+Publish+Container+Images+to+GAR).
6+
57
Note that this method intended for rapid prototyping rather than for production workloads.
68
If you need to transition a prototype to a production deployment,
79
[file a Data Platform and Tools > Operations bug](https://bugzilla.mozilla.org/enter_bug.cgi?product=Data+Platform+and+Tools&component=Operations)

‎src/cookbooks/publishing_datasets.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ labels:
2222
schedule: daily # scheduled in Airflow to run daily
2323
public_json: true
2424
public_bigquery: true
25-
review_bug: 1414839 # Bugzilla bug ID of data review
25+
review_bugs: # Bugzilla bug IDs of data review
26+
- 1414839
2627
incremental_export: false # non-incremental JSON export writes all data to a single location
2728
```
2829
@@ -39,7 +40,7 @@ The following options define how data is published:
3940
- `incremental`: indicates how data gets updated based on the query and Airflow configuration
4041
- `true`: data gets incrementally updated
4142
- `false`: the entire table data gets updated
42-
- `review_bug`: Bugzilla bug number to the data review
43+
- `review_bugs`: list of Bugzilla bug numbers of the data review
4344

4445
Data will get published when the query is executed in Airflow. Metadata of available public
4546
data on Cloud Storage is updated daily through a separate Airflow task.

‎src/datasets/glam.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ GLAM data is also meant to be explored by itself: GLAM aggregation tables are us
66

77
## GLAM final tables (Aggregates dataset)
88

9-
The following datasets are split in three categories: Firefox Desktop Glean, Firefox Desktop Legacy and Firefox for Android. The tables contain the final aggregated data that powers GLAM.
9+
The following datasets are split in three categories: Firefox Desktop Glean, Firefox Desktop Legacy and Firefox for Android. The tables contain the final aggregated data that powers GLAM. The `moz-fx-data-shared-prod.glam_etl` tables listed below are views of the authoritative tables in the `moz-fx-glam-prod.glam_etl` dataset.
1010

1111
Each link below points to the dataset's page on [Mozilla's Data Catalog](https://mozilla.acryl.io/) where you can find the dataset's full documentation.
1212

‎src/datasets/other/sponsored_tiles/sponsored_tiles.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,6 @@ Note: gotchas with historical Sponsored Tiles revenue data are outlined [here](h
2020
| All Mozillians | `telemetry.newtab` | Firefox Desktop > `Newtab` | Expanded `newtab` **desktop** dataset. Requires unnesting events. |
2121
| All Mozillians | `telemetry.newtab_interactions` | Firefox Desktop > New Tab Interactions | In-development dataset for basic analyses. Available metrics are limited to **desktop** clicks and impressions. |
2222
| Contextual Services | `contextual_services.event_aggregates` | Contextual Services > Event Aggregates | Dataset with Sponsored Tiles and Suggest analyses by advertiser. No longer maintained. Instead use the derived `event_aggregates_[product]` datasets. |
23-
| Contextual Services | `contextual_services.event_aggregates_spons_tiles` | Contextual Services > Event Aggregates `Spons` Tiles | Workhorse dataset for Sponsored Tiles analyses by advertiser. |
23+
| Contextual Services | `contextual_services.event_aggregates_spons_tiles` | Contextual Services > Event Aggregates `Spons` Tiles | Being deprecated; superseded by `Newtab` and ad metrics datasets. Dataset for Sponsored Tiles analyses by advertiser. |
2424
| Contextual Services | `contextual_services_derived.adm_forecasting` | Contextual Services > `AdM` Forecasting | Dataset with required components for Sponsored Tiles and Suggest revenue forecasts. |
2525
| Contextual Services, Revenue | `contextual_services.event_aggregates` x `revenue.revenue_data` | Revenue > `AdM` Revenue with Telemetry | Revenue information combined with usage metrics. This dataset is useful for `CPC` analyses. |

‎src/datasets/pings.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,10 @@ for the ping (default to 1000 per process, governed by a
6565

6666
It was introduced in Firefox 62.
6767

68+
Note that this describes the legacy "event" ping. Glean's "events" ping, which newer
69+
applications use, is instead sent when the application goes to background or a
70+
certain number of events is reached.
71+
6872
### "update" ping
6973

7074
**Obsolete since Firefox 157**
@@ -112,7 +116,8 @@ It was introduced in Firefox 72, replacing the ["optout" ping][optout_ping]
112116
### "coverage" ping
113117

114118
The [coverage ping][coverage_ping] ([announcement](https://blog.mozilla.org/data/2018/08/20/effectively-measuring-search-in-firefox/))
115-
is a periodic census intended to estimate telemetry opt-out rates.
119+
was a periodic census intended to estimate telemetry opt-out rates.
120+
It is no longer sent; the ping was removed from Firefox in [bug 1773060](https://bugzilla.mozilla.org/show_bug.cgi?id=1773060).
116121

117122
We estimate that [93% of release channel
118123
profiles](https://docs.google.com/document/d/1EnQoq9o1sLXTgsbbG8mPnUiWIOmKSINGJOAKyYUm_YA/edit#)
@@ -191,4 +196,4 @@ To augment our data collection, see [Collecting New Data][addprobe] and the
191196
[addprobe]: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html
192197
[datacollection]: https://wiki.mozilla.org/Firefox/Data_Collection
193198
[pingsender]: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/internals/pingsender.html
194-
[coverage_ping]: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/coverage-ping.html
199+
[coverage_ping]: https://github.com/mozilla-firefox/firefox/blob/4c877370d5cb831951ab634bacaa065598ec8d70/toolkit/components/telemetry/docs/data/coverage-ping.md

‎src/introduction/tools.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,9 @@ Note that while STMO is not yet considered deprecated, Looker is the preferred s
3838

3939
## Deprecated tools
4040

41-
These tools are still available, but are generally not recommended.
41+
These tools have been retired and are no longer available.
4242

4343
### Telemetry Measurement Dashboard
4444

45-
The [Telemetry Measurement Dashboard](https://telemetry.mozilla.org/new-pipeline/dist.html) (TMO) site is the 'venerable standby' of Firefox telemetry analysis tools.
46-
It is the predecessor to GLAM (see above) and is still lightly maintained until we are sure that GLAM covers all of its use cases.
45+
The [Telemetry Measurement Dashboard](https://telemetry.mozilla.org/new-pipeline/dist.html) (TMO) site was the 'venerable standby' of Firefox telemetry analysis tools.
46+
It is the predecessor to GLAM (see above). As of December 5, 2024, the data system underpinning it was turned off entirely; use GLAM instead.

‎src/metrics/metrics.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ This section contains definitions and information about standard metrics used at
1414

1515
Daily Active Users or DAU counts the number of unique profiles active in the product on each day. This is intended to approximate the number of people using the product each day.
1616

17-
The most accurate and up-to-date metric information can be found on the
17+
Supporting metric information can be found on the
1818
[Metrics Inventory](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/620494911/Metrics+Inventory) page in Confluence (Mozilla LDAP required).
1919

20-
The official source code definitions for the metrics can be found in
20+
The canonical, official source code definitions for the metrics can be found in
2121
[metric-hub](https://github.com/mozilla/metric-hub/tree/main/definitions) on GitHub.
2222

2323
### Details
@@ -42,7 +42,7 @@ If the number of users stays constant, but the average number of active profiles
4242

4343
This measure attempts to tell us whether, after creating a profile, the user returned at any point in the next 6 days. This is meant to measure whether a product is successful in engaging users early on: at present, many users churn after the first run and we want to measure whether efforts to make them stick around are succeeding.
4444

45-
This metric is used for all mobile applications, excluding Fenix and Firefox Preview, and is a top-level OKR inside Mozilla for 2020. It can apply both on the level of a specific product (e.g. Firefox iOS, Lockwise for Android) as well as an aggregate measure across all mobile products and devices. This metric is _not_ used for Firefox desktop.
45+
This metric is used for all mobile applications, excluding Fenix and Firefox Preview, and was a top-level OKR inside Mozilla in 2020. It can apply both on the level of a specific product (e.g. Firefox iOS, Lockwise for Android) as well as an aggregate measure across all mobile products and devices. This metric is _not_ used for Firefox desktop.
4646

4747
### Details
4848

0 commit comments

Comments
 (0)