Skip to content

Commit d217342

Browse files
Copilotcfc4n
Copilot
and
cfc4n
authored
docs: add AI agent methodology for Android BoringSSL offset generation
Add ai/memory/android-boringssl-offset-generation.md documenting: - Step-by-step process for generating new Android BoringSSL offsets - Struct comparison methodology between Android versions - Common pitfalls (InplaceVector stride, private fields, PAC vs offset) - Verification checklist for offset correctness - File reference table for all related source files Agent-Logs-Url: https://github.com/gojue/ecapture/sessions/8eb6baee-d22c-40f5-8548-ea35a5c07838 Co-authored-by: cfc4n <709947+cfc4n@users.noreply.github.com>
1 parent 084908f commit d217342

1 file changed

Lines changed: 276 additions & 0 deletions

File tree

‎ai/memory/android-boringssl-offset-generation.md‎

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# Android BoringSSL Offset Generation Methodology
2+
3+
## Purpose
4+
5+
This document describes the step-by-step methodology for generating eBPF struct offset headers when a **new Android version** ships with a **new BoringSSL** that changes internal struct layouts. eCapture's TLS key capture on Android depends on hardcoded byte offsets into BoringSSL's private C++ structs (`ssl_st`, `ssl_session_st`, `bssl::SSL_HANDSHAKE`, `bssl::SSL3_STATE`, etc.). When Google updates BoringSSL for a new Android release, those offsets may shift and must be regenerated.
6+
7+
> **Important**: Do NOT reference `utils/boringssl_non_android_offset.sh` — that script is for non-Android (upstream) BoringSSL and has different struct layouts and branches.
8+
9+
---
10+
11+
## Prerequisites
12+
13+
- Linux x86_64 build host with `g++`, `git`
14+
- Access to `https://android.googlesource.com/platform/external/boringssl`
15+
- Repository cloned at project root (`go.mod` present)
16+
17+
---
18+
19+
## Step 1: Clone the Android BoringSSL Repository
20+
21+
```bash
22+
# From project root
23+
BORINGSSL_REPO=https://android.googlesource.com/platform/external/boringssl
24+
BORINGSSL_DIR="./deps/boringssl"
25+
git clone ${BORINGSSL_REPO} ${BORINGSSL_DIR}
26+
```
27+
28+
Reference: `utils/boringssl_android_offset.sh` lines 1–20.
29+
30+
---
31+
32+
## Step 2: Check Out the Target Android Release Branch
33+
34+
Android BoringSSL branches follow the naming convention `android${VERSION}-release`:
35+
36+
```bash
37+
cd ${BORINGSSL_DIR}
38+
git fetch --tags
39+
git checkout android16-release # or android17-release, etc.
40+
```
41+
42+
Also check out the **previous** version's branch (e.g., `android15-release`) to compare structs.
43+
44+
---
45+
46+
## Step 3: Identify Struct Differences
47+
48+
The key file is `src/ssl/internal.h`. Compare it between the two branches:
49+
50+
```bash
51+
# From deps/boringssl
52+
git diff android15-release..android16-release -- src/ssl/internal.h
53+
```
54+
55+
### Structs to Check
56+
57+
| Struct | Header Reference | Key Fields for eCapture |
58+
|---|---|---|
59+
| `ssl_st` | `src/ssl/internal.h` | `version`, `session`, `rbio`, `wbio`, `s3` |
60+
| `ssl_session_st` | `src/ssl/internal.h` | `ssl_version` (or `secret_length` in older), `secret`, `cipher` |
61+
| `bssl::SSL3_STATE` | `src/ssl/internal.h` | `hs`, `client_random`, `exporter_secret`, `established_session` |
62+
| `bssl::SSL_HANDSHAKE` | `src/ssl/internal.h` | `new_session`, `early_session`, `hints`, `client_version`, `state`, `tls13_state`, `max_version`, and TLS 1.3 secret fields |
63+
| `bio_st` | `src/ssl/internal.h` or `include/openssl/bio.h` | `num`, `method` |
64+
| `bio_method_st` | `include/openssl/bio.h` | `type` |
65+
| `ssl_cipher_st` | `src/ssl/internal.h` | `id` |
66+
67+
### Common Breaking Changes to Watch For
68+
69+
1. **Field type changes**: e.g., `uint8_t secret_[48]` → `InplaceVector<uint8_t, 48>` (49 bytes instead of 48)
70+
2. **Field removal**: e.g., `hash_len_` removed, `ssl_st.version` removed, `secret_length` removed
71+
3. **Field addition**: e.g., new `ssl_version` field in `ssl_session_st`
72+
4. **Field reordering**: Any reordering shifts all subsequent field offsets
73+
5. **Type size changes**: e.g., `InplaceVector` adds a `size_` byte per field
74+
75+
---
76+
77+
## Step 4: Create the Offset Source File
78+
79+
### If the struct layout is **compatible** with the previous version
80+
81+
Use the existing `utils/boringssl-offset.c` (the generic Android offset calculator).
82+
83+
### If the struct layout has **breaking changes**
84+
85+
Create a new dedicated offset file, e.g., `utils/boringssl-android16-offset.c`.
86+
87+
The offset source file uses the C++ `offsetof()` macro against BoringSSL headers:
88+
89+
```c
90+
#include <openssl/base.h>
91+
#include <openssl/crypto.h>
92+
#include <ssl/internal.h>
93+
#include <stddef.h>
94+
#include <stdio.h>
95+
96+
// Define X-macro lists for standard and changed fields
97+
#define SSL_STRUCT_OFFSETS \
98+
X(ssl_st, session) \
99+
X(ssl_st, rbio) \
100+
// ... etc
101+
102+
#define X(struct_name, field_name) \
103+
format(#struct_name, #field_name, offsetof(struct struct_name, field_name));
104+
SSL_STRUCT_OFFSETS
105+
#undef X
106+
```
107+
108+
Key considerations:
109+
- If `ssl_st.version` was removed, do NOT include `X(ssl_st, version)` in the list
110+
- If `ssl_session_st.secret_length` was replaced by `ssl_session_st.ssl_version`, update accordingly
111+
- For `InplaceVector`-based fields, add them as separate offsetof() entries since they are now public fields
112+
113+
Compile and run:
114+
```bash
115+
g++ -Wno-write-strings -Wno-invalid-offsetof \
116+
-I include/ -I . -I ./src/ offset.c -o offset
117+
./offset
118+
```
119+
120+
---
121+
122+
## Step 5: Generate the Kern Header File
123+
124+
The offset program's stdout produces `#define` directives. Wrap them in a header guard:
125+
126+
```bash
127+
HEADER_FILE="kern/boringssl_a_${VERSION}_kern.c"
128+
echo "#ifndef ECAPTURE_BORINGSSL_A_${VERSION}_KERN_H" > ${HEADER_FILE}
129+
echo "#define ECAPTURE_BORINGSSL_A_${VERSION}_KERN_H" >> ${HEADER_FILE}
130+
./offset >> ${HEADER_FILE}
131+
```
132+
133+
### Handle Version-Specific Differences
134+
135+
For versions with breaking changes, add sentinel defines BEFORE the `#include` directives:
136+
137+
```c
138+
// If secret_length field was removed:
139+
#define SSL_SESSION_ST_SECRET_LENGTH 0xFF
140+
141+
// If TLS 1.3 secrets use InplaceVector instead of raw arrays:
142+
#define BORINGSSL_INPLACEVECTOR_SECRETS
143+
144+
// If hash_len_ field was removed (read from InplaceVector.size_ instead):
145+
#define BSSL__SSL_HANDSHAKE_HASH_LEN (BSSL__SSL_HANDSHAKE_SECRET+0x30)
146+
```
147+
148+
Then add the standard includes:
149+
```c
150+
#include "boringssl_const.h"
151+
#include "boringssl_masterkey.h"
152+
#include "openssl.h"
153+
154+
#endif
155+
```
156+
157+
---
158+
159+
## Step 6: Update boringssl_const.h (if needed)
160+
161+
`kern/boringssl_const.h` derives TLS 1.3 secret offsets from the `max_version` offset.
162+
163+
If the new Android version changes how secrets are laid out (e.g., InplaceVector), add a conditional:
164+
165+
```c
166+
#ifdef BORINGSSL_INPLACEVECTOR_SECRETS
167+
// Use pre-computed offsets from the kern header directly
168+
#define SSL_HANDSHAKE_SECRET_ BSSL__SSL_HANDSHAKE_SECRET
169+
#define SSL_HANDSHAKE_EARLY_TRAFFIC_SECRET_ BSSL__SSL_HANDSHAKE_EARLY_TRAFFIC_SECRET
170+
// ... etc
171+
#else
172+
// Original computation from max_version offset
173+
#define SSL_HANDSHAKE_HASH_LEN_ roundup(BSSL__SSL_HANDSHAKE_MAX_VERSION+2,8)
174+
#define SSL_HANDSHAKE_SECRET_ SSL_HANDSHAKE_HASH_LEN_+8
175+
// ... etc
176+
#endif
177+
```
178+
179+
---
180+
181+
## Step 7: Update boringssl_masterkey.h (if needed)
182+
183+
Check that `kern/boringssl_masterkey.h` handles:
184+
- Reading TLS version from the new location (e.g., `SSL_SESSION_ST_SSL_VERSION` instead of `SSL_ST_VERSION`)
185+
- Fixed master key length when `secret_length` is removed (use `BORINGSSL_SSL_MAX_MASTER_KEY_LENGTH`)
186+
- Hash length from InplaceVector's `size_` field instead of `hash_len_`
187+
188+
Use `#ifdef` guards for version-specific code paths, keyed on defines from the kern header (e.g., `SSL_SESSION_ST_SSL_VERSION`).
189+
190+
---
191+
192+
## Step 8: Update boringssl_android_offset.sh
193+
194+
Add the new Android version to the `sslVerMap` in `utils/boringssl_android_offset.sh`:
195+
196+
```bash
197+
sslVerMap["4"]="16" # android16-release
198+
```
199+
200+
Add conditional logic to use the dedicated offset source file:
201+
202+
```bash
203+
if (( val > 15 )); then
204+
cp -f ${PROJECT_ROOT_DIR}/utils/boringssl-android16-offset.c ${BORINGSSL_DIR}/offset.c
205+
else
206+
cp -f ${PROJECT_ROOT_DIR}/utils/boringssl-offset.c ${BORINGSSL_DIR}/offset.c
207+
fi
208+
```
209+
210+
---
211+
212+
## Step 9: Verify
213+
214+
1. **Offset program compiles and runs**:
215+
```bash
216+
cd deps/boringssl && git checkout android${VERSION}-release
217+
cp ../utils/boringssl-android${VERSION}-offset.c offset.c
218+
g++ -Wno-write-strings -Wno-invalid-offsetof -I include/ -I . -I ./src/ offset.c -o offset
219+
./offset # Should print #define lines
220+
```
221+
222+
2. **Generated kern header matches expected offsets**: Cross-reference offsets with `offsetof()` calculations manually or via a test program.
223+
224+
3. **Go build succeeds**:
225+
```bash
226+
go build ./...
227+
```
228+
229+
4. **Go tests pass**:
230+
```bash
231+
go test ./internal/... ./pkg/... ./cli/...
232+
```
233+
234+
---
235+
236+
## Offset Verification Checklist
237+
238+
When verifying offsets, check these critical fields:
239+
240+
- [ ] `ssl_st->s3` — pointer to SSL3_STATE (used to chain to handshake state)
241+
- [ ] `ssl_st->session` — pointer to ssl_session_st (TLS 1.2 master secret)
242+
- [ ] `bssl::SSL3_STATE->hs` — pointer to SSL_HANDSHAKE (TLS 1.3 secrets)
243+
- [ ] `bssl::SSL3_STATE->client_random` — 32 bytes, must be correct for keylog
244+
- [ ] `bssl::SSL3_STATE->exporter_secret` — TLS 1.3 exporter
245+
- [ ] `bssl::SSL_HANDSHAKE->state/tls13_state` — handshake state machine values
246+
- [ ] `bssl::SSL_HANDSHAKE->secret` through `expected_client_finished` — TLS 1.3 secret fields (stride matters!)
247+
- [ ] `ssl_session_st->secret` — TLS 1.2 master secret data
248+
- [ ] `ssl_session_st->cipher` — cipher suite pointer
249+
250+
---
251+
252+
## Common Pitfalls
253+
254+
1. **Do not use `boringssl_non_android_offset.sh`** — it targets upstream BoringSSL from `boringssl.googlesource.com`, which has a completely different struct layout than Android's fork at `android.googlesource.com`.
255+
256+
2. **InplaceVector stride is 49, not 48** — When BoringSSL replaces `uint8_t[48]` with `InplaceVector<uint8_t, 48>`, each field is 49 bytes (48 data + 1 byte `size_`). Using 48-byte stride will cause cumulative offset drift.
257+
258+
3. **Private fields cannot use `offsetof()` directly** — BoringSSL's TLS 1.3 secret arrays are in a `private:` section. In older versions, compute them relative to `max_version`. In newer versions with InplaceVector, they may be public and `offsetof()` works.
259+
260+
4. **PAC (Pointer Authentication Code) is separate from offsets** — On ARM64 Android 16+ devices, pointers may contain PAC signature bits in the upper bits. This is handled by `STRIP_PAC()` in `kern/common.h` and is independent of struct offset correctness.
261+
262+
5. **Always download and analyze actual source code** — Never guess offsets. Always clone the BoringSSL branch and compile the offset program against its headers.
263+
264+
---
265+
266+
## File Reference
267+
268+
| File | Purpose |
269+
|---|---|
270+
| `kern/boringssl_a_${VER}_kern.c` | Per-Android-version offset defines |
271+
| `kern/boringssl_const.h` | Computes TLS 1.3 secret offsets from base offset |
272+
| `kern/boringssl_masterkey.h` | eBPF program that reads TLS secrets using offsets |
273+
| `utils/boringssl-offset.c` | Generic Android offset calculator (≤A15) |
274+
| `utils/boringssl-android16-offset.c` | Android 16+ offset calculator (InplaceVector) |
275+
| `utils/boringssl_android_offset.sh` | Orchestration script for offset generation |
276+
| `kern/common.h` | STRIP_PAC macro for ARM64 pointer authentication |

0 commit comments

Comments
 (0)