Organizations are now leveraging cloud-native capabilities for stability, scalability, accuracy, and speed in their applications. They are modernizing their legacy systems by creating automated, AI-enabled DataOps platforms. One major challenge is the time spent on data preparation, validation, and accuracy, leading to increased costs and lower data quality.
Wipro, an AWS Premier Consulting Partner and Managed Service Provider (MSP), addresses these challenges by delivering cloud transformation solutions using DataOps with GenAI.
GenAI in AWS
AWS Bedrock is a fully managed GenAI service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, DeepSeek, Luma, Meta, Mistral AI, poolside (coming soon), Stability AI, and Amazon through a single API. It provides a broad set of capabilities needed to build generative AI applications with security, privacy, and responsible AI. Using Amazon Bedrock, you can easily experiment with and evaluate top FMs for your use case, privately customize them with your data using techniques such as fine-tuning and Retrieval Augmented Generation (RAG), and build agents that execute tasks using your enterprise systems and data sources.
AWS Bedrock can be used to generate code in various stages of the software development lifecycle (SDLC). allows developers to create their own systems to augment, write, and audit code by using models within Amazon Bedrock instead of relying on out-of-the-box coding assistants.
Code interpretation in Amazon Bedrock enables your agent to generate, run, and troubleshoot your application code in a secure test environment. This includes tasks such as understanding user requests for specific tasks, generating code to perform those tasks, executing the code, and providing the result from the code execution.

Solution Overview-
The solution we have proposed is using Anthropic Claude 3 sonnet model which will generate the automated scripts based on the prompt provided by the user.
Why Claude-3-sonnet model: -
It has the ability to perform nuanced content creation, accurate summarization and handle complex scientific queries. This model demonstrates increased proficiency in non-English languages and coding tasks with more accurate responses, supporting a wider range of use cases on a global scale.

image describes the architectural components

Data Flow-
1) User will fill the table in excel requesting different inputs formats for different services to be created as part of DataOps Pipeline. This excel will then be uploaded to S3 bucket.
2) The event-based architecture triggers an event as S3 push to call the respective Lambda function to start the bedrock invocation process.
3) Bedrock model will be invoked using boto3 – invoke_model as shown below:

Image describes the code snippet

4) Response from the bedrock will be used for creation of template for script deployment or directly as an executable script.
5) Lambda will use the generated template for creation of Jobs.

The various capabilities of our solution are:-

  • Ability to effectively perform the script generation activities using AWS native GenAI services.
  • Increased Reusability of the script used for result generation from the models.
  • Auto optimized script generation from GenAI.
  • Cost Effective solution based on serverless architecture.
  • DataOps driven automated framework will provide fully integrated skeleton for reuse.
  • Efficiency and effectiveness in script preparation.
  • Event based trigger for pipeline creation and processing.
  • Tight coupling with all AWS native services.
  • Less manual intervention in the fully integrated solution.
  • End to End data delivery using cloud agnostic solution provide scalability and cost effectiveness.

Benefits:

  • Process Efficiency: Increases overall efficiency in script generation process upto 90%
  • Effort optimization: Up to 30-40% reduction in involvement of in-house teams required for data preparation activities.
  • Reduction in the requirement for proficient and highly skilled people.

Industrial usage-
The DataOps with GenAI as a solution has benefits across industries as efficient data preparation is required by most of the industry process for operational functions. E.g., for Manufacturing industry monthly sales analysis, for health care it could be medical records used for future prediction of upcoming health challenges, for Media industry finding out the TRP driven content in real time, etc. So the overall solution will deliver cloud transformation at scale with GenAI in a speedy manner needed for most of the organizations and implementing the solution leveraging Amazon Cloud services which offers the benefit of Quick Win, optimal cost and unlimited scalability.