Contributes to the design of information infrastructure, and data management processes to move the organization to a more sophisticated, agile and robust target state data architecture
Develop systems that ingest, cleanse and normalize diverse datasets, develop data pipelines from various internal and external sources and build structure for previously unstructured data
Develop good understanding of how data will flow & stored through an organization across multiple applications such as CRM, Broker & Sales tools, Finance, HR etc
Design & develop data management and data persistence solutions for application use cases leveraging relational, non-relational databases and enhancing our data processing capabilities
Develop POCs to influence platform architects, product managers and software engineers to validate solution proposals and migrate
What we are looking for:
Bachelor’s degree in Information Science, Computer Science, Mathematics, Statistics or a quantitative discipline in science, business, or social science.
Minimum of 3 years of experience as a data developer using Python, Kafka, Spark Streaming, Azure SQL Server, Cosmos DB/Mongo DB, Azure Event Hubs, Azure Data Lake Storage, Azure Search etc.
Excellent technical, analytical and organizational skills.
Effective written and verbal communication skills, including technical writing.
Hands-on engineering lead who is curious about technology, should be able to quickly adopt to change and one who understands the technologies supporting areas such as Cloud Computing (AWS, Azure(preferred), etc.), Micro Services, Streaming Technologies, Network, Security etc
Hands-on Experience for building Data Pipelines in Cloud.
Experience in working with databases especially SQL server databases.
Design & develop data management and data persistence solutions for application use cases leveraging relational, non-relational databases and enhancing our data processing capabilities.
Experience handling un-structured data, working in a data lake environment, leveraging data streaming and developing data pipelines driven by events/queues
Team player, Reliable, self-motivated, and self-disciplined individual capable of executing on multiple projects simultaneously within a fast-paced environment working with cross functional teams
Required Skills:
Apache Spark, Data Pipelines, Python, Spark Streaming