feat(embedding): update to Google Generative AI model gemini-embedding-001 and refactor embedding logic - #32
Conversation
…ng-001` and refactor embedding logic
hoangsonww
added
bug
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
✅ Deploy Preview for mern-stack-ecommerce-website ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the application's embedding infrastructure by transitioning to Google's Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request is a solid refactoring that successfully migrates to the new gemini-embedding-001 model and centralizes embedding logic into a new embeddingService. This greatly improves code structure and maintainability. The changes are applied consistently, and the new service is well-tested. I have a couple of suggestions to further improve maintainability and test isolation.
| beforeAll(() => { | ||
| process.env.GOOGLE_AI_API_KEY = 'test-key'; | ||
| }); | ||
|
|
||
| beforeEach(() => { | ||
| mockEmbedContent.mockReset(); | ||
| }); |
There was a problem hiding this comment.
Modifying process.env within tests can cause side effects that interfere with other test suites running in the same process. To ensure test isolation, it's best practice to restore the original environment variable's value after the tests in this suite have completed. You can achieve this by storing the original value before the suite runs and restoring it using an afterAll block.
const originalApiKey = process.env.GOOGLE_AI_API_KEY;
beforeAll(() => {
process.env.GOOGLE_AI_API_KEY = 'test-key';
});
afterAll(() => {
process.env.GOOGLE_AI_API_KEY = originalApiKey;
});
beforeEach(() => {
mockEmbedContent.mockReset();
});| @@ -0,0 +1,79 @@ | |||
| require('dotenv').config(); | |||
There was a problem hiding this comment.
Calling require('dotenv').config() inside a service module can introduce side effects and make configuration management less predictable, as behavior can depend on the order in which modules are imported. It's a common best practice to handle environment configuration at the application's entry point (e.g., in index.js or a dedicated config.js file) to ensure variables are loaded once and consistently across the entire application.
There was a problem hiding this comment.
Pull request overview
This PR migrates the backend embedding pipeline from the deprecated Google model to Gemini (models/gemini-embedding-001) by introducing a centralized embeddingService and refactoring existing Pinecone/FAISS/Weaviate sync scripts/services to use it for consistent 768-dim normalized vectors.
Changes:
- Added
backend/services/embeddingService.jsto centralize embedding request construction, dimension enforcement (768), and normalization. - Refactored Pinecone sync/service code and FAISS/Weaviate scripts to use
embedText(...)with retrieval task types. - Added Jest unit tests for
embeddingServiceand updated docs to reflect the new embedding model.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| backend/services/embeddingService.js | New shared service for Gemini embeddings (request building, normalization, validation). |
| backend/tests/embeddingService.spec.js | Unit tests covering request shape, normalization, and dimension mismatch errors. |
| backend/sync/syncPinecone.js | Uses embeddingService for embeddings during bulk Pinecone sync. |
| backend/services/pineconeSync.js | Uses embeddingService for per-product Pinecone sync hooks/services. |
| backend/scripts/build-faiss-index.js | Uses embeddingService to generate vectors and sets FAISS index dimension to 768. |
| backend/scripts/search-faiss-index.js | Uses embeddingService to embed queries before FAISS search. |
| backend/scripts/weaviate-upsert.js | Uses embeddingService for embeddings during Weaviate upsert. |
| README.md | Updates docs to reference gemini-embedding-001. |
| ARCHITECTURE.md | Updates architecture diagram text to gemini-embedding-001. |
| AGENTS.md | Updates agent documentation references to gemini-embedding-001. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const trimmedText = text?.trim(); | ||
| if (!trimmedText) return null; | ||
|
|
There was a problem hiding this comment.
embedText returns null for blank/whitespace input. Since most callers expect an array of 768 floats, this can lead to downstream runtime errors if a whitespace string slips through. Consider throwing a descriptive error for empty input (or returning a consistent empty array) and updating callers accordingly so the contract is unambiguous.
| const trimmedText = text?.trim(); | |
| if (!trimmedText) return null; | |
| if (typeof text !== 'string') { | |
| throw new TypeError('embedText: `text` must be a non-empty string.'); | |
| } | |
| const trimmedText = text.trim(); | |
| if (!trimmedText) { | |
| // Return a consistent zero vector when input is blank/whitespace | |
| return new Array(EMBEDDING_DIMENSION).fill(0); | |
| } |
| // 2) Embed the query text | ||
| const genAI = new GoogleGenerativeAI(GOOGLE_AI_API_KEY); | ||
| const model = genAI.getGenerativeModel({ model: 'models/text-embedding-004' }); | ||
| const embedding = (await model.embedContent(query)).embedding.values; | ||
| const embedding = await embedText(query, { | ||
| taskType: EMBEDDING_TASK_TYPES.RETRIEVAL_QUERY, | ||
| }); |
There was a problem hiding this comment.
embedText(query) can return null when the CLI argument is only whitespace (because embedText trims internally). In that case index.search(embedding, k) will throw. Consider validating query.trim() before calling embedText, or handle a null return with a clear error/exit path.
This pull request migrates the application's embedding pipeline from the deprecated Google
text-embedding-004model to the new Geminigemini-embedding-001model. It introduces a centralizedembeddingServicefor generating, normalizing, and validating embeddings across all vector database integrations (Pinecone, FAISS, Weaviate). The update ensures consistent use of 768-dimensional normalized vectors, simplifies embedding code, and adds robust error handling and test coverage for embedding generation.Embedding Model Migration
text-embedding-004togemini-embedding-001, reflecting the new default for generating product embeddings. [1] [2] [3] [4]Centralized Embedding Service
backend/services/embeddingService.jsto encapsulate Gemini embedding logic, including model initialization, request building, vector normalization, and error handling for embedding size mismatches.pineconeSync.js,syncPinecone.js,build-faiss-index.js,search-faiss-index.js,weaviate-upsert.js) to use the newembeddingServicefor generating embeddings, removing direct usage of the Google Generative AI SDK. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]Testing and Validation
embeddingServiceto verify request construction, embedding normalization, and error handling for incorrect embedding sizes.Documentation Updates
Codebase Simplification
Let me know if you have any questions about the new embedding workflow or how to use the
embeddingService!