Job ID:
J55295
Job Title:
Senior AI Observability & Evaluation Engineer
Location:
Dallas, TX
Duration:
15 Months + Extension
Domain Exposure:
Manufacturing
Client:
To Be Discussed Later
Employment Type:
W-2, 1099, C2C
Work Authorization:
US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other
Job Title:Senior AI Observability & Evaluation Engineer
Location:Dallas, TX
Duration: 15 Months + Extension
Bill Rate: $78/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
Job Description
We are looking for an experienced AI Observability and Evaluation Engineer with strong expertise in Python, Generative AI, LLM monitoring, and production AI systems. The ideal candidate should have hands-on experience implementing observability frameworks, evaluating LLM outputs, monitoring RAG pipelines, and improving the reliability and performance of AI-powered applications. The candidate should understand how to measure AI quality, identify hallucinations, and monitor agentic AI workflows in production environments.
Required Skills
6+ years of experience in software engineering, Python, or AI/ML engineering.
Strong hands-on experience with Python and REST APIs.
Experience working with LLMs and Generative AI applications.
Strong knowledge of OpenTelemetry and distributed tracing.
Experience with LangSmith, Langfuse, or MLflow.
Knowledge of RAG pipelines, embeddings, and vector databases.
Experience with AI evaluation and performance monitoring.
Knowledge of Prometheus, Grafana, and monitoring tools.
Experience with cloud platforms such as AWS, Azure, or GCP.
Strong understanding of API monitoring, logging, and debugging.
Experience with Docker, Kubernetes, and CI/CD pipelines.
Recent / Modern Skills (2025–2026)
LLM observability and agent tracing.
RAG evaluation using RAGAS, DeepEval, or TruLens.
AI quality assessment and hallucination detection.
Prompt evaluation and version management.
Agentic AI workflow monitoring.
MCP instrumentation and monitoring.
Token usage, latency, and cost tracking.
AI guardrails and safety evaluation.
OpenTelemetry-based LLM tracing.
Production AI reliability and performance optimization.
Responsibilities
Design and implement observability solutions for Generative AI applications.
Monitor LLM performance, latency, token usage, and application costs.
Develop evaluation frameworks for RAG and agentic AI applications.
Track hallucinations, response quality, and model reliability.
Implement distributed tracing using OpenTelemetry.
Build dashboards and monitoring solutions using Grafana and Prometheus.
Troubleshoot production AI issues and improve application reliability.
Collaborate with AI engineers, platform teams, and architects.
Establish evaluation metrics and quality standards for enterprise AI applications.
We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.