This project is currently inactive. You can reapply when the client reopens it and begins accepting applications. Dismiss
This website uses cookies to ensure you get the best experience on our website. By continuing to use this site, you agree to our cookie  & privacy policy.Accept
search

Select a Skill

Talent

Join as Talent Post a project

checked This is a sample alert

logo

Data Engineer-GenAI Chatbot Data Pipelines

  • Full Time
  •  On-Site -
  • 6 months
  • Project posted on: 28/03/2024

Data Engineer-GenAI Chatbot Data Pipelines

  • Information Technology
  • DBA/datawarehousing
  • Technology Operations and Support
  • Data Science

On-Site -

Full Time

6 months

 28/03/2024

Assignment Details

Our client, a leading global specialist in energy management and automation is looking to engage with a Data Engineer-GenAI Chatbot Data Pipelines
1. Data Ingestion and Pipeline Design
Data Sources:
Ingest data from diverse sources relevant to GenAI chatbots.
Handle structured (e.g., databases, APIs) and unstructured (e.g., text, images) data.
Integration with Chatbot Architecture:
Understand the GenAI chatbot architecture (e.g., RAG, LLMs)
Design pipelines that seamlessly integrate with chatbot components.
Critical understanding of Vector DB, Indexting Strategies, various chunking strategies.
2. Configurable and Reusable Pipelines
Configurability:
Create pipelines with configurable parameters.
Allow customization for different chatbot products.
Common Components:
Develop reusable building blocks (e.g., connectors, transformations).
Ensure consistent data flow across various chatbot instances.
3. Data Validation and Quality Assurance
Validation Checks:
Implement data validation during ingestion.
Detect anomalies, missing values, and inconsistencies.
Quality Metrics:
Define quality metrics (accuracy, completeness, timeliness).
Monitor data quality over time.
4. Gap Analysis and Improvement
Gap Identification:
Analyze gaps in data availability.
Identify missing data points or discrepancies.
Root Cause Investigation:
Collaborate with domain experts to understand data gaps.
Propose solutions for improving data completeness.
5. Analytics and Reporting
Data Insights:
Build analytics around data pipelines.
Extract trends, patterns, and performance metrics.
Reporting Mechanisms:
Create dashboards or reports for stakeholders.
Communicate pipeline health and data quality effectively.

Skills Required

1. Proficiency in data ingestion from various sources, including structured and unstructured data.
2. Familiarity with GenAI chatbot architecture (e.g., RAG, LLMs) to design pipelines that seamlessly integrate with chatbot components.
3. Knowledge of Vector DB, indexing strategies, and chunking strategies for efficient data handling.
4. Ability to create configurable pipelines with parameters that can be customized for different chatbot products.
5. Experience in developing reusable building blocks such as connectors and transformations to ensure consistent data flow across various chatbot instances.

About the Client

A leading global specialist in energy management and automation

Industry

IT Services & Consulting

Minimum Experience:

5+ years

Additional Skills:

  • Information Security Management System
  • DBMS
  • Infrastructure System Integrations
  • Data Warehousing
  • Embedded Systems Design
  • New Product Development
  • Instructional Design

No of open positions:

1

How it works

  • 1

    Build your
    skill profile

  • 2

    Strengthen it
    with feedback

  • 3

    Determine your
    optimal fee level

  • 4

    Apply for matching
    projects

  • 5

    Get finalized

  • 6

    Stay Engaged

Build your skill prodile

Project
step1
shape_step

Request feedback from prior projects and/or work stints

shape_step

Determine your optimal fee level

  • starFeeBee helps consultants understand how to price their services.
  • starEnter your skill & location to get ranges for what similar professionals have charged on projects.
  • starUnlock more details e.g., best paying sub-skills, ranges by projects durations, by simply adding a benchmark from a orior projects.
graph_step2
shape_step

Apply for matching projects

shape_step

Get finalized!

Project
step5
shape_step

Stay Engaged!

Project
step6
shape_step

Similar Freelance Projects you can apply to.

Consultant- Embedded Tester

IT Services & Consulting

  • Software Development - C/C#/.NET
  • Software Development - web technologies
  • Quality Assurance & Testing
  • Technology Operations and Support
  • Software architect
clock_icon 6 month(s)
exp_icon5+ years
location_icon On-Site
SKILLS REQUIRED
  • #GitHub
  • #Python
  • #JIRA
  • #ABAP
  • #automation testing
  • #ADC
Apply

Consultant Tableau Developer

IT Services & Consulting

  • Software Development - web technologies
  • DBA/datawarehousing
  • Business Analysis
  • UI/UX
  • Technology Management
clock_icon 3 month(s)
exp_icon7+ years
location_icon Hybrid
SKILLS REQUIRED
  • #Presentation Skills
  • #Tableau developer
  • #ETL
  • #Analytic development
  • #ETL Development
  • #Advanced Excel
Apply

Consultant—Product Manager (Software Quality & Assurance)

Software Products & Services

  • Project/Program Management
  • Quality Assurance & Testing
  • Technology Operations and Support
  • Product Management
  • Technology Management
clock_icon 3 month(s)
exp_icon5+ years
location_icon Remote
SKILLS REQUIRED
  • #Product Management
  • #QA methodologies
  • #Software Quality Assurance
  • #Software Development Lifecycle
  • #Project Management
Apply

Consultant - Android IN Developer

IT Services & Consulting

  • Software Development - web technologies
  • Technology Operations and Support
  • Technology Management
clock_icon 12 month(s)
exp_icon10+ years
location_icon Remote
SKILLS REQUIRED
  • #Android
  • #MVVM
  • #ABAP
  • #Mobile Apps Program Management
  • #C#
  • #APIs
Apply

Consultant—Software Engineer (Python and AWS)

IT Services & Consulting

  • Project/Program Management
  • Technology Operations and Support
  • Information Security
  • Software architect
  • Technology Management
clock_icon 3 month(s)
exp_icon4+ years
location_icon Hybrid
SKILLS REQUIRED
  • #Python
  • #Software Engineer
  • #AWS
  • #Kubernetes
  • #Azure
  • #Docker
Apply

Consultant - Lead Software engineer(Automation Testing)

IT Services & Consulting

  • Software Development - C/C#/.NET
  • Software Development - Mainframe
  • Software Development - web technologies
  • Quality Assurance & Testing
  • Technology Operations and Support
clock_icon 3 month(s)
exp_icon6+ years
location_icon Hybrid
SKILLS REQUIRED
  • #Python
  • #AWS
  • #Oracle
  • #Automation Testing
  • #C#
  • #SQL
Apply

Lead Cloud Engineer Consultant (Azure, Python)

IT Services & Consulting

  • Software Development - web technologies
  • Technology Operations and Support
  • Software Development - Python
clock_icon 3 month(s)
exp_icon7+ years
location_icon Hybrid
SKILLS REQUIRED
  • #Python
  • #cloud engineer
  • #Debugg
  • #AWS
Apply

IT Expert in CRM: Keynote Speaker (Manufacturing/Healthcare/Tech)

FMCG

  • ERP/CRM
  • Technology Operations and Support
  • CIO/CTO functions
  • Technology Management
clock_icon less than 1 month(s)
exp_icon15+ years
location_icon Remote
SKILLS REQUIRED
  • #CIO / CTO
  • #Leadsquare CRM
  • #Speaker
Apply

Consultant Azure DevOps

IT Services & Consulting

  • Software Development - web technologies
  • Technology Operations and Support
  • Software Development - Python
clock_icon 3 month(s)
exp_icon7+ years
location_icon On-Site
SKILLS REQUIRED
  • #Python
  • #Azure Cloud services
  • #CI/CD tools
  • #Kubernetes
  • #Cluster deployment
  • #Azure DevOps
Apply

Consultant - Data Bricks

IT Services & Consulting

  • Software Development - web technologies
  • Technology Operations and Support
  • Technology Management
clock_icon 3 month(s)
exp_icon6+ years
location_icon On-Site
SKILLS REQUIRED
  • #Python
  • #Kafka
  • #Azure Databricks
  • #Azure Blob Storage
  • #data architectures
  • #SQL
Apply

Consultant—Test Engineer (API and Microservices)

IT Services & Consulting

  • Software Development - web technologies
  • Quality Assurance & Testing
  • Technology Operations and Support
  • Information Security
  • Software architect
clock_icon 4 month(s)
exp_icon4+ years
location_icon On-Site
SKILLS REQUIRED
  • #Pen Testing
  • #Microservices
  • #Security Testing
  • #APIs
  • # Test Management
  • #RESTful
Apply

Consultant—Manual Tester (Windchill PLM Software)

IT Services & Consulting

  • Quality Assurance & Testing
  • Technology Operations and Support
  • Software architect
  • Technology Management
clock_icon 6 month(s)
exp_icon4+ years
location_icon On-Site
SKILLS REQUIRED
  • #SAP
  • #ERP systems
  • #CAD tools
  • #AutoCAD
  • #Software Quality
  • #Agile methodologies
Apply
View More Projects btn btn

More than 3000 clients rely on Flexing It today

item_1
item_2
item_3
item_4
item_5
item_6
item_8
item_9

Frequently asked questions.

We host both full-time & part-time projects from top clients for 100+ skill categories. We also help connect experts like you with clients for short consultation calls called Flexperts

You just need to create a ‘boutique firm’ account during the registration process. The rest of the application process remains the same.

Our freelancers fee benchmarking tool, FeeBee helps our freelancers and consultants find out what their peers charged for similar projects. Built on 600K+ verified benchmarks, registered users get unlimited access to it.

As a freelancing platform, we do not provide permanent positions or full-time job to our consultant network.

Flexing It will be involved end to end right from contracts to invoicing. Once you are finalized, your payment details are transparently laid out in your contract terms with your client and our team actively liaises with the client finance team to ensure timely payments.

We do not charge any fee or commission from you. Your fee quote will be paid out to you in full as per your project contract.