Assignment Details
Our client, a leading global specialist in energy management and automation is looking to engage with a Data Engineer-GenAI Chatbot Data Pipelines
1. Data Ingestion and Pipeline Design
Data Sources:
Ingest data from diverse sources relevant to GenAI chatbots.
Handle structured (e.g., databases, APIs) and unstructured (e.g., text, images) data.
Integration with Chatbot Architecture:
Understand the GenAI chatbot architecture (e.g., RAG, LLMs)
Design pipelines that seamlessly integrate with chatbot components.
Critical understanding of Vector DB, Indexting Strategies, various chunking strategies.
2. Configurable and Reusable Pipelines
Configurability:
Create pipelines with configurable parameters.
Allow customization for different chatbot products.
Common Components:
Develop reusable building blocks (e.g., connectors, transformations).
Ensure consistent data flow across various chatbot instances.
3. Data Validation and Quality Assurance
Validation Checks:
Implement data validation during ingestion.
Detect anomalies, missing values, and inconsistencies.
Quality Metrics:
Define quality metrics (accuracy, completeness, timeliness).
Monitor data quality over time.
4. Gap Analysis and Improvement
Gap Identification:
Analyze gaps in data availability.
Identify missing data points or discrepancies.
Root Cause Investigation:
Collaborate with domain experts to understand data gaps.
Propose solutions for improving data completeness.
5. Analytics and Reporting
Data Insights:
Build analytics around data pipelines.
Extract trends, patterns, and performance metrics.
Reporting Mechanisms:
Create dashboards or reports for stakeholders.
Communicate pipeline health and data quality effectively.
Skills Required
1. Proficiency in data ingestion from various sources, including structured and unstructured data.
2. Familiarity with GenAI chatbot architecture (e.g., RAG, LLMs) to design pipelines that seamlessly integrate with chatbot components.
3. Knowledge of Vector DB, indexing strategies, and chunking strategies for efficient data handling.
4. Ability to create configurable pipelines with parameters that can be customized for different chatbot products.
5. Experience in developing reusable building blocks such as connectors and transformations to ensure consistent data flow across various chatbot instances.
Share
6 month(s)
5+ years 
Remote
